Z440 BIOS problem with PLX PEX8311 PCI Express-to-Local Bus Bridge

[fgorseth]

We are experiencing the same problem with our PCIe x1 boards (which we are developing inhouse).

Our boards are also using PEX8311.

We are experiencing this problem with HP ProLiant DL380 Gen9, but our PCIe boards are working perfectly fine with DL380 Gen8.

We also experience the same problem with ASUS X99-A motherboards, which are using the same chipset as the HP ProLiant DL380 Gen8 server: Intel X99.

[JochemBonarius]

Hi,

We are delevoping PCIe boards for Digital Broadcasting industries, which tend to use HP ProLiant servers for their reliability. A number of our customers have recently bought a new HP Gen9 server, only to find out that their cards will not work any longer. We have asked our customers to get ticket numbers for us. Here's two already: 4651502848 and 4745803770

Small summary: all our cards using the PLX (now rebranded to Avago) PEX811x chip will not work correctly on HP ProLiant Gen9 servers. These are cards that have been operating without problems in many systems worldwide for almost 10 years. It seems Avago has received numerous complaints from customers regarding this issue and are analyzing it. It turns out the HP bios is configuring the PCIe card incorrectly (i.e. not according to PCIe specification). HP acknowledged this and is issuing a bios update, which is to be released in August.

In the meantime Avago suggested a workaround that prevents the most problematic PCI configuration error. However, the workaround does not seem to work in all cases: still many of their customers –including us- are having issues getting PEX811x cards running on HP ProLiant Gen9 servers. Again: these issues have never occurred on any other system in many years, so the problem is likely caused by HP.

In the meantime I am in contact with Avago regarding this issue. Although they are also still in the dark about the root cause of this issue, we are making small steps at a time. We are trying to find the exact point where the card’s operation becomes unstable. As expected, this now seems to occur during (HP) BIOS configuration. By using post-boot PCI configuration changes, we have managed to get a card working correctly. But this only seems to work in less than 50% of the start-ups: sometimes the card already locks-up at boot.

As soon as we find the root cause, we will probably quickly have a workaround. Then we will hope HP will fix it in its BIOS for a more robust solution; It really seems that the issue in being caused by the HP ProLiant Gen9 bios: HP seems to confirm that, hence the bios update.

with regards,

Jochem Bonarius

[JochemBonarius]

Hi all,

It is awfully quiet here...

We have an update: apart from the issue described in the first posts, the BIOS makes a second error in interpreting the PCIe capabilities. PCI configuration register 0x64h (PCIe Device capabilities) reports 000b in bits 2:0. This indicates the Maximum Payload Size Supported is 128 bytes.

According to our PCIe protocol analysis captures, the HP BIOS reads this register. Furhtermore, it writes-back this value to PCI configuration register 0x68h (PCIe Device Control): it sets bits 7:5 to 000b, indicating a Maximum Payload Size of 128 bytes.

However, in further protocol analysis we see that the Root Complex (PCH) seems to ignore this setting (or is configured incorrectly): When doing for instance an Uplink MemRead (End point reading data in the main memory through DMA), the Root Complex replies with Completion TLP's with a payload up to 256 bytes! This corrupts the data and causes the PEX to respond with an ERR_FATAL.

As a test we reduced the PEX Programmed Prefetch Size from 512 bytes to 128 bytes, which seemed to solve the lock-up issue. However, this reduces our maximum throughput with 40%, which is not sufficient for many of our applications. So even though the cards is operating now, its performance is not sufficient.

We really need this fixed in the BIOS. Please help

[MSLEEE]

Again, please escalate the case through the HP Support so that HP can validate the problem and can provide bootleg BIOS until the official BIOS will be released.

[JochemBonarius]

Hi MSLEEE,

Our customers have already made tickets for us: 4651502848 and 4745803770.
How can we get some support? Please help.

With regards,

Jochem Bonarius

[MSLEEE]

As long as I undetstood, this forum is for discussions on Workstations. This is NOT the official process to contact HP Support. Small number of HP employees will be updating voluntarily by their knowledge.  If there is a bug in HP product or you want to get the official answer or solution, I believe that you have to contact HP Support.

[MSLEEE]

It looks like the case # 4651502848 is for ProLiant and case # 4745803770 is not the valid number.  As long as I understand, one of two people you mentioned below emailed you yesterday and asked you to send the card to replicate the problem. 

[Stephen6]

Yes, the workstation firmware team is aware of this issue. This problem has been reproduced and corrected.  I'm not sure when the change will make it to the field, but thought that I would update all concerned on the progress.  Please continue to follow MSLEE's instructions.

I am an HP Employee.
My opinions are my own, and do not express those of HP.

[igork]

Hi

We purchased a Digium TE420 card with PLX PEX8112 chip. It's not recognized by BIOS. Tried different BIOS settings.  Will HP fix this in future?

Server: HP ProLiant DL380 Gen9

BIOS: 1.40_05-06-2015

Regards

[Dan_WGBU]

HP will release an updated Z840/Z640/Z440 BIOS in September 2015, that will have changes to fix these PLX831x issues on these Workstations.