Z840 PCIe3 Slot Query For NVMe Expansion Card

[ooms]

Z840 Bios 2.6.2, 1 x E5-2680 V4 (120W),  8 x 16GB - DDR4 2400MHz, 850w PSU.

Hi. I'm looking to buy an ASUS Hyper M.2 x16 PCIe 3.0 x4 V2 which supports 4 x NVMe M.2 but I'm struggling with whether that's feasible with my current setup. I'd like to stick to a single CPU, for power consumption reasons, so that rules out one of the PCIe3x16 slots where the ASUS card could go. If I must go dual-CPU, I've read other posts where it's inadvisable to run two 120w+ CPUs on the 850w power supply so I'd have to swap the CPU I have now, plus, more power draw overall.  I need the 10GbE SFP+ card for throughput reasons so don't know if I have a way forward. 

PCIe3 x4 – CPU0  Single NvME Drive through adapter
PCIe3 x16 – CPU0 Graphics Card
PCIe3 x8 – CPU1 Unavailable
PCIe3 x16 – CPU1 Unavailable
PCIe2 x8 (4) PCH (single proc)/ PCIe3 x8 CPU1 (dual proc)  Empty
PCIe3 x16 – CPU0  Mellanox ConnectX-3 EN Pro 10GbE SFP+
PCIe2 x1 – PCH Empty

Is anyone aware of a graphics card that could go in the top PCIe3 x4 slot thus freeing up the PCIe3 x16? Alternatively, I don't fully understand what PCIe2 x8 (4) PCH means for that currently empty slot. Could the Mellanox card go in there instead?

Appreciate any insight or should I just bite the bullet and pick up 2 lower power CPUs? Thanks.

[DGroves]

i recommend the dual cpu route as it increases the value and performance of the system

besides the power supply a second cpu and matching  ram (for the sec cpu mem slots) will be required and like the power supply these parts can be sourced used on ebay quite cheaply

the cost for a used 1200 watt z840 power supply is not that great usually 70 to around 150.00 USD

https://www.ebay.com/itm/384622377613

[ooms]

Thanks for the reply. DGroves. I'm going to pick up some lower-wattage CPUs and drop those in as changing the power supply is not something I want to attempt myself after watching Art of Server's video! This is just for a homelab so power consumption is a big consideration but this is likely the easiest route to get the M2 card into this server. Thanks!

[DGroves]

Huh???!!!!.....................not sure what you watched but changing the power supply in the z 820/840 systems is performed by simply removing the power cord and pulling on the side cover  latch then grabbing the black handle on the power supply and pulling it out.. reinstalling installing the new power supply is done by inserting it into the  power supply area and pushing it in till flush with the case edges...................... this procedure takes between 20sec  to 2 min

[SDH]

To the OP:

Some added info below. I'd listen to DGroves... he has been working with these for years now on an enterprise level.Two full capacity PCIe3 x16 slots if one processor.

No, you don't lose use of one of the two full power PCIe x16 (mechanical and electrical) slots if you only run one processor. See the diagram above, left column, from the HP Z840 Technical White Paper (LINK  ) .

HP specifies that if you want to run 4 NVMe M.2 sticks in the Z840 you put the proper PCIe x16 card in the bottom x16 "video" slot. Plus, you also generally need to go into BIOS and change that slot's bifurcation setting from Auto to x4x4x4x4. Keeping it at Auto generally works in the ZX G4 generation of workstations, however. Each M.2 stick needs 4 true electrical lanes to run at full speed. You don't want to run them on reduced lanes or in a lower than PCIe3 generation slot. Bifurcation changes the x16 PCIe3 electrical lanes to four PCIe3 x4 sets. A lot of this info came out after the manuals for the ZX40 workstations were created and NVMe M.2 capabilities were enabled by BIOS upgrades. My posts here can get you some added HP literature now not easily found. Here's a ZX40 service manual link worth having: LINK   Of interest you don't get PCIe4 slots until the ZX G5 family of HP workstations, but we found PCIe4 NVMe M.2 sticks actually do run faster than PCIe3 M.2 sticks in the ZX40 PCIe3 generation of workstations.

Do you really need/want 4 sticks? What about 2? Why the "ASUS Hyper M.2 x16 PCIe 3.0 x4 V2", which has only passive cooling? Why not consider the HP-engineered Z Turbo Drive Dual Pro (version 1 or 2, both of which will run one or two PCIe4 M.2 sticks even if the card is in a PCIe3 x16 or PCIe3 x8 slot? If you really need 4 M.2 sticks consider instead the HP Z Turbo Drive Quad Pro for about 85.00 USD from eBay, lightly used. You run that in your lower PCIe3 x16 slot and can run one, two, three, or four M.2 sticks in it. The cable shown below transfers M.2 activity signals out to a special 2-pin header on the motherboard... not a big deal but you don't get that engineering from the ASUS:

Note active cooling capability...

I've posted here in the forum details about the ZTD Dual Pro (versions 1 and 2) and the ZTD Quad Pro... easy to find with the search bar above to the left.

"I don't fully understand what PCIe2 x8 (4) PCH means": Look at the HP info in the two columns in their 1 vs 2 processor(s) table above more closely. They explain how a PCIe2 x4 active-electrical-lanes slot (but with x8 mechanical shape) can be transformed into a PCIe3 x8 active-electrical-lanes slot (still with its x8 mechanical shape). They're activating a second set of x4 true lanes that were not active before the second processor went in.  Move Mellanox card to there after upgrades to the power supply and 2nd processor. My recall is you don't need a superslot to run 10Gbe at full speed... your Mellanox ConnectX-3 EN Pro 10GbE SFP+ card only needs a PCIe3 x8 electrical lanes slot which you'll get for a grand total of $14.00 (plus the power supply cost).

I agree with DGroves... get the better power supply and a second processor to match the first. Your E5-2680 v4 has a single sSpec code of SR2N7 and is going for $14.00 USD (!) now here on US eBay. And consider getting a ZTD Quad Pro. I got one of those to experiment with in our Z440 and Z640 builds, its fan is very quiet, and it turns out we don't need 4 M.2s. For our needs I like the ZTD Dual Pro v1 best.

[ooms]

Oh really? Not sure if this link will make it through but this was what I was going off. https://www.youtube.com/watch?v=HKSc5l0cvsw&t=1009s

[SDH]

I looked that up too, know the guy's work, and the source is respected as is DGroves. I posted a detailed response to your questions on the other issues and some moderator (or artificial ignorance algorithm) deleted it as SPAM. Nothing of the sort and I've asked higher levels to repost it properly.

The basic items are:

1. Having one processor is all that is needed for you to be able to use both PCIe x16 slots at full speed. An identical second processor will cost you as low as $14.00 USD from eBay here (!).

2. Adding a second identical processor changes that PCIe2 x4 slot you've mentioned to a PCIe3 x8 slot, through the magic of HP engineering. Thus you can put your fast network card in that slot and run it at full speed if you add the processor.

3. I really think you should buy a HP Z Turbo Drive Quad Pro if you really need 4 M.2 sticks. Those in great condition are about 85.00 now on eBay. They have a very quiet active cooling fan, unlike the ASUS. You can run them with 1,2,3, or 4 sticks.

4. To use 2 or more M.2 sticks in the card in your bottom PCIe x16 slot in a Z440/Z640/Z840 you have to turn on bifurcation in BIOS for that slot to x4x4x4x4. It is easy. Each M.2 stick only needs x4 electrical lanes. Thus a true x16 slot can run up to 4 M.2 sticks at full speed.

5. I figured this all out for our company... and have found that the newer PCIe4 M.2 sticks also work fine in the HP PCIe3 based workstations like your Z840. They actually run faster than a PCIe3 stick running in the same card/ same slot so that is the type we buy now even for our ZX40 and ZX G4 workstations (both only PCIe3 technology). You don't get to PCIe4 slots until the HP ZX G5 workstations.

6. I've done a number of posts here on the secrets of the Z Turbo Drive PCIe cards... pretty easy to find. HP came out with a bunch of knowledge about this after the main manuals were written. There are good HP documents worth finding. If you only need 2 M.2 sticks consider the Z Turbo Drive Dual Pro (versions 1 and 2) which is mainly what we use now.

7. No spam here... frustrates me because my first post was better and took me significant time to create for you.

[SDH]

The complaint to HP moderator(s) resulted in posting of my original, above. We stand ready to help you more if you wish.

[DGroves]

a very few "EARLY" x840 systems may have shipped without the sec cpu's gpu power cable, in all my years i've never came across one  i suspect these systems are quite rare and as i recall the HP official sec cpu upgrade kit only came with the cpu/heatsink and no upgraded power supply cable

and FYI the missing connector is for the upgraded supplies SECOND GPU POWER CONNECTOR,  unless you need this for a high wattage video card that has dual gpu connectors the z820/840 system will still work fine with dual cpu's and the 1200 watt supply

you can simply pull the existing power supply and look inside the area where the power supply connectors are and note if you are missing this connector

[ooms]

Really appreciate all the info. The items below are on their way to me now!

1) HP Z840 Workstation 1125W Power Supply Unit 719799-003
2) HP Z840 Z820 Professional Workstation CPU Cooling Heatsink 749598-001
3) HP Z Turbo Drive M.2 NVME Quad Pro Adapter PN: 804086-003 841969-001
4) Intel Xeon E5-2680 V4 (SR2N7) x 1
5) HP (809082-591) - 16GB PC4-19200T-R (1RX4, DDR4-2400MHz) ECC REG RAM x 8