Z440 BIOS problem with PLX PEX8311 PCI Express-to-Local Bus Bridge

[AGruber]

Hello,

in our company we are using a peripheral card with the PLX PEX8311 chip in Z420 workstations with RHEL 6.5.

We are currently planning the transition from Z420 to Z440 and we noticed that the card is not working with the Z440.

Technical details:

- HP Z440, Intel Xeon E5 -1630 v3, 16GB Memory

- Red Hat Enterprise Linux 6.5

- PLX PEX8311 PCI Express-to-Local Bus Bridge: Vendor: 0x10b5, Device: 0x8111

At first we tried different slots and BIOS settings but it didn't solve the problem.

We already contacted the Avagotech (formerly PLX) support which told us following:

"On boot the BIOS is incorrectly writing to 88h (PECS_MAINDATA) with the value 0 which is why the 1000h (PECS_DEVINIT) register is incorrectly being cleared as the default value of 84h (PECS_MAININDEX) is 0."

This causes the PCI part of the card "behind" the PEX 8311 bridge not being initialized correctly.

We verified that the same configuration of Hardware/Software is working without problems in a Z420.

Therefore we can confirm this behaviour by comparing the "lspci -xxxx" output on a Z420 and a Z440.

So we tried to modify the device driver such that the mentioned register is getting written during driver loading process. Unfortunately we were not able to correct the issue with this workaround.

Is this a known issue and is it going to be addressed in upcoming BIOS releases in the near future?

It is crucial to us getting these cards working in the Z440 workstations.

Best regards,

Andy Gruber

[jonds]

Hello,

we have the same issue using an HP ProLiant DL380 Gen9 server (both test with old bios v1.21 and latest v1.32)
Our PCIe board also uses the PLX PEX8311 chip.

My investigation reveiled that the BIOS writes only bit5 at offset 0x88 (PECS_MAINDATA) of the PCI Config space of the bridge device to zero. (to do that it probably performs a read, modifies bit5, and next writes back the result).

By default PECS_MAININDEX (0x84) is normally at zero, so bit5 of PECS_DEVINIT (0x1000) gets cleared, resulting in blocking of all DMA traffic from our card towards the PC.

As a temporarily test workaround I've changed the default value of the PECS_MAININDEX register to 0x30 (by modifying the PLX EEPROM on our card). Now bit5 of the unused mailbox register (0x1030) is destroyed by the HP BIOS, but this doesn't cause any harm for our board. (you can also use the eeprom to fill the mailbox register with 0xFFFFFFFF and after the bios has run, you can read 0xFFFFFFDF back from this register, clearly showing the problem)

Is someone from HP actually reading this forum and could this be forwarded to the BIOS developers ?

Many thanks,

Jon

[Dan_WGBU]

What specific peripheral card are you using?  Is it available from a third party, or is it a proprietary card? 

[LouisShay]

Hello HP Support,

I'm a factory Applications Engineer  from Avago Technologies (formerly PLX).  

Over the past couple months, we've been fielding a number of support calls from our customers who manufacture PCIe adapter cards based on PEX8311, PEX8112  bridge devices.    (dozens of vendors, 100's of boards, so we expect to see many more such reports going forward.)   Z440 is one of a handful of platforms affected.  

We suspect Z440 BIOS assumes that 8311 presents a Type2 PCIe capabilities header.  8311/8112 present a Type 1 PCIe capabilties header.  In the Type2 PCie capabilities structure, config  register offset 88h  is DeviceControl2.  Bit 5 of this register, per the PCIe spec, is reserved 0).   We see BIOS come and clear this bit at enumeration time. 

For the 8311, 8112 bridge,  however, config register offset 88h is something else.  When BIOS clears this bit, the adapter card can no longer generate DMA accesses to PCIe.  

That's one issue,  but you can see from some of these posts that even restoring 88h[5] bit doesn't completely fix the problem for some boards.  

I will gladly make myself available to assist with debugging this problem.

Regards,

Louis Shay

Avago Technologies

[jonds]

Hi Dan,

the card is a card developed by our company and which is sold all over the world in industrial applications.

Many of our customers are using HP ProLiant servers and were not affected (e.g. ProLiant DL380 Gen8).

Only recently since the use of ProLiant DL380 Gen9, the issue popped up.

If needed we could provide such a card to HP if this would speed up the BIOS fixing.

Best regards,

Jon

[LouisShay]

Small clarification:  

where I said "PCIe Type 2 ",   I meant to say PCIe Capabilituy version 2,  which is for  PCIe 2.0 & 3.0 devices (v1 = PCIe 1.x).  The 8x11/8112 are still treated as PCI Type 1 header.

-LS

[LouisShay]

Hello PEX8311, PEX 8112 Developers, 

There appears to be a solution using the 8311's SPI eeprom at boot time.   In the SPI EEPROM,  modify the value of the register MAININDEX  (PCI config register 84h=0x30).   This will re-direct BIOS access to config register 88h away from DEVINIT,  and instead to MAILBOX0, where the BIOS write is harmless. 

Regards, 

Louis Shay

Avago Technology

[Dan_WGBU]

Andy, Jon, and Lewis, I sent each of you a message about this problem.  Apparently I cannot send an email to multiple recipients, so I sent 3 messages.  If each of you did not get a message, let me know. 

Andy, HP will look at your problem. 

Jon, is this a different board than Andy's?  Are you using ProLiant servers only? 

Louis, HP will appreciate your input. 

Let's take the technical discussions offline.  I sent you my contact info in each of the messages.

Thanks,

Dan

[MSLEEE]

Spoiler

 Please call HP Support and update here with the CASE # information so that HP can follow up properly.

[MSLEEE]

Please call HP Support and update here with the CASE # information so that HP can follow up properly.