Multi-M.2 NVMe PCIe card in Z240

[fmyhr]

Hi,

I'd like to use an HP Z Turbo Drive Quad Pro or other multiple-M.2 NVMe PCIe card like a Supermicro AOC-SLG3-2M2 in a Z240 workstation with Xeon E3-1245v5. I gather that for this to work the Z240 needs to support PCIe bifurcation, i.e. running the x16 PCIe slot as four separate x4 links. Z440/Z640/Z840 BIOS history shows PCIe bifurcation support was added sometime prior to April 2017, with BIOS 2.22. But AFAICT Z240 never got a similar BIOS update. I believe the Z240 hardware is capable of supporting PCIe bifurcation, it just needs a BIOS that enables it. For instance, the similar LGA 1151 Supermicro X11SSL motherboard supports PCIe bifurcation.

It appears technically possible for an advanced/adventurous Z240 user to modify the BIOS to enable PCIe bifurcation . It would be far preferable if HP could release an official BIOS with this support.

I'm curious if anyone has tried using such a card with a Z240?

Thanks,

Frank

[DGroves]

the z240 is a low end budget system and  as far as i know, its chipset/hardware will not fully support bifurcation

there are 3rd party quad SSD carriers available that do not use bios based bifurcation, they do it on the quad ssd cards hardware but these cards are not cheap costing  well over $500.00 just for the card  and for that price you are well on your way to buying a z440 system

note the card below requires a x16 slot and is non bootable

https://www.sonnettech.com/product/m2-4x4-pcie-card.html

https://amfeltec.com/pci-express-gen-3-carrier-board-for-m-2-ssd/

and there are others search for them using google

[fmyhr]

@DGroves,

Thanks for your reply.

Low cost, high efficiency and low power usage (14W idle!) are reasons I love HP Z2x0. I am not aware of any reason the Z240 hardware cannot support PCIe bifurcation. AFAIK it's a feature the CPU provides when properly configured by BIOS. The same E3-1245v5 CPU provides this feature in the Supermicro motherboard I linked. As ixsystems user Stux says :

---

Bifurcation is essentially a bios setting. For it to work you should have retimer chips on the multi-card adapter*. No need for a full PCIe switch.

---

The cards you linked** use a hardware PCIe switch / PLX to run multiple PCIe links behind the motherboard's single link. This is a more flexible solution than PCIe bifurcation -- I think it allows cases with different numbers of PCIe lanes on either side of the switch. And, at least for the M.2 carrier cards that hardwire the PLX's configuration, requires no BIOS support. But as you note PLXes are expensive, and they use additional power (notice the heatsinks and fans on the chips). Using a PLX-based carrier card for PCIe bifurcation is a waste of money and power if the CPU (with proper BIOS configuration) can already do it.

So I remain interested if anyone has tried (or decides to try!) using PCIe bifurcation with a Z240. I think the only slot that can do it is the x16 slot connected to the CPU. And I think the BIOS needs to be modified to enable this support*** (see link in first post.) I might try it myself if I find time. Will the HP "Sure Start" backup BIOS prevent bricking the motherboard if I flash a modified BIOS that turns out not to work? I now believe that HP Sure Start will not allow any non-official-HP BIOS to work, and that its strong protection / restriction cannot be turned off. So BIOS that supports PCIe bifurcation will have to come (if ever) from HP.

Thanks,

Frank

* The Supermicro AOC-SLG3-2M2 card (for one) does include retimer chips.

** Thanks for the link to the amfeltec card, I wasn't aware of that one. For reference, the dual M.2 Syba SI-PEX40129 is the least expensive carrier card with PLX that I've found.

*** No longer sure of this. It may be possible (using Linux, at least) to issue a PCIe hot reset and reconfigure the x16 slot as bifurcated. I will post more details, and perhaps try this myself, later.

---

[fmyhr]

I did some more digging.

• A good technical introduction to PCIe bifurcation:

  https://blogs.synopsys.com/expressyourself/2016/10/14/flashback-to-basics-friday-bifurcation/

• The Intel 100 Series / C230 Series Chipset datasheet Vol 1 p22 shows that the Z240's C236 PCH does support PCIe bifurcation.
• The Intel Xeon E3-1200V5 datasheet Vol 1 p24 lists the supported PCIe link configurations of the PEG (= PCIE Express Graphics) x16 slot driven by the CPU: 1x16, 1x16 Reversed, 2x8, 2x8 Reversed, 1x8 + 2x4, and 1x8 + 2x4 Reversed.

So the Z240 CPU x16 PCIe port cannot support 4x4 bifurcation -- in other words it is incapable of supporting four NVMe drives unless you use a carrier card that has a PCIe switch. But it definitely supports 2x8 or 1x8 + 2x4 bifurcation, which would allow using 2 or 3 NVMe drives in this slot, no PLX required.

[DGroves]

your last comment does not take into account how a board maker has configured the motherboard's  existing slot configuration or available pci-e slots

in other words, the z240's current pci-e lane setup uses almost all of the available pci-e lanes and would not allow bifurcation on the remaining x16 slot (the primary x16 is used for a video card) without either disabling a slot or severely limiting a existing x8 slot,... to do so without any problems would most likely require additional board logic such as a plx chip

[fmyhr]

Indeed the Z240 has only 1 slot that is x16 electrically. This is the slot driven by CPU, and the one that supports PCIe bifurcation when so configured. The Z240's other physical x16 slot is x4 electrically, and driven by the PCH -- I do not think that slot supports bifurcation even if you wanted (say) 4x1 lanes instead of the existing 1x4.

You are correct that the x16 slot is often used for a video card. However, some Z240s like mine have CPUs with integrated graphics, which leaves the x16 slot free. Also, it is possible (at least with Z230, haven't tried with Z240) to use a PCIe x1 video card in one of the other slots driven by the PCH.

[DGroves]

and again i will point out you gloss over the z240's board design, it lacks the required logic/hardware to perform bifurcation

adding a bios option for bifurcation will do nothing unless the motherboards logic can support it

rerouting pci-e lanes requires such logic and while some of it is indeed in the intel chipset, it also requires supporting chips (such as a plx chip) to fully implement it