Inconsistent Values When Reading and Writing PCI Registers

Introduction

A quick intro to the topic at hand.

In some technical processes, reading from and writing to registers may lead to inconsistencies or invalid values due to the simultaneous execution of read and write operations. This article provides an overview of this issue and offers potential solutions for troubleshooting it.

Problem

If gathersing the pci information from /sys (sysfs) the data shows pci link speed and status. In some cases, the LnkSta that is returned indicates the following "16GT/s(downgraded) x4(downgraded)". While 16GT/s and x4 might be the correct speed and width, the ouput shows "downgraded".

$ lspci | grep downgr
...
        LnkSta: Speed 8GT/s (downgraded), Width x8 (ok)
        LnkSta: Speed 16GT/s (ok), Width x8 (downgraded)
...

This could match the following criteria

• Inconsistencies in data when comparing multiple reads from the same register
• Observing incorrect or unexpected values that do not align with the expected behavior.
• Frequent occurrences of invalid values (e.g., once or twice a week).

Resolution

While this is not a resolution to the problem, this could help gathering information that you can bring to your hardware provider.

Below is the lspci code that emits the string: “(downgraded)”. Just so we know what criteria is being used here:

static char *link_compare(int type, int sta, int cap)
{
  if (sta > cap)
    return " (overdriven)";
  if (sta == cap)
    return "";
  if ((type == PCI_EXP_TYPE_ROOT_PORT) || (type == PCI_EXP_TYPE_DOWNSTREAM) ||
      (type == PCI_EXP_TYPE_PCIE_BRIDGE))
    return "";
  return " (downgraded)";
}

This is called from: cap_express_link()

  w = get_conf_word(d, where + PCI_EXP_LNKSTA);
  sta_speed = w & PCI_EXP_LNKSTA_SPEED;
  sta_width = (w & PCI_EXP_LNKSTA_WIDTH) >> 4;
  printf("\t\tLnkSta:\tSpeed %s%s, Width x%d%s\n",
        link_speed(sta_speed),
        link_compare(type, sta_speed, cap_speed),
        sta_width,
        link_compare(type, sta_width, cap_width));

The link_compare() code came from this discussion:

In sum, it’s checking the status speed from PCI_EXP_LNKSTA_SPEED and if it’s less than cap_speed then it’s always reporting “(downgraded)” for devices other than PCI_EXP_TYPE_ROOT_PORT, PCI_EXP_TYPE_DOWNSTREAM, PCI_EXP_TYPE_PCIE_BRIDGE.

The PCI configuration data, including speed and width information that lspci code reads and potentially reports as “downgraded,” is pulled from the sysfs entries under /sys/bus/pci/devices. As an example, for NVMe devices, the data is accessed from the corresponding entries for each NVMe device under /sys/bus/pci/devices/[DEVICE_ID]/config.

The reading of PCI configuration data is managed by the kernel's PCI subsystem, specifically within the PCI sysfs interface and related functions. The primary functions involved include:

pci_read_config() in drivers/pci/pci-sysfs.c
pci_user_read_config_##size in drivers/pci/access.c

These functions handle the reading of configuration data directly from the hardware registers of the PCI devices.

If you use a script that reads these data, is good to understad that it is just getting raw data out of the device on the PCI bus.

For debugging you could try using dmesg and other system logs to check for any hardware events or errors reported by the kernel that might indicate link renegotiation or other PCIe issues.

Look for AER (Advanced Error Reporting) messages which can sometimes provide clues about PCIe link issues:

$ dmesg | grep AER

Possible Causes

1. Simultaneous read and write operations on the same register: This can result in capturing an intermediate state instead of the final value, leading to inconsistencies.
2. Incorrect timing between read and write operations: If there's not enough time between reading and writing, it may lead to the capture of an incorrect or invalid value.
3. Hardware malfunctions: Faulty hardware components or software glitches can cause unexpected behavior in register values.

References & related articles

lspci: Don't report PCIe link downgrades for downstream ports Accessing PCI device resources through sysfs