Z840 PCiE Bifurcation for M.2 NVMe

[Fiendish]

Hi all, I want to add some M2 storage to my Z840, probably four drives, to use for video editing.

I've seen some old  threads about this which were very useful, such as this one...

https://h30434.www3.hp.com/t5/Business-PCs-Workstations-and-Point-of-Sale-Systems/Z840-PCiE-Bifurcat...

I was looking at the Sabrent adapter cards which seem to have good cooling. They have two options, the EC-P3X4 with onboard bifurcation, and the EC-P4BF which passes through to use the mobo's bifurcation support.

https://www.amazon.co.uk/dp/B0BRYQH443/ref=twister_B0C4396BBK?_encoding=UTF8&th=1

https://sabrent.com/collections/memory-and-storage/products/ec-p4bf

My question is which is better on the Z840? The EC-P3X4 onboard model is available in the UK, whereas the pass through model is only available on import here for some reason. The EC-P3X4 also only requires at x4 slot which is handy for my setup.

Also can anyone confirm that the Z840 is PCIe Gen 3?

many thanks,

Ollie

[SDH]

Regarding PCIe Gen 3.... yes, any slot you'd want to use are Gen 3. Thats all in the technical/service manual. You want to search for 748721-004 to get the latest one, look on page 56. That workstation has 3 PCIe x16 slots, all PCIe Gen 3. Only 2 slots are Gen 2 and one of those is a legacy PCI slot (not PCIe).

I'd upgrade to the latest BIOS released April this year. I'd personally only choose to get the HP Z Turbo Drive Quad Pro for running 4 sticks. You can find those cards without sticks, used. They are built like tanks, to enterprise grade, with banks of capacitors to provide backup power for proper onboard saving of data if there is a sudden power outage. HP provides quite a bit of very specific info on best slot positioning... I'm sure you've found those PDFs. For 2 NVMe sticks I chose to buy a ZTD Dual Pro and it does not have the banks of capacitors the Quad Pro does. I have quality UPS power backup.

To see the speed of the ZTD QP card on a non-HP motherboard that is VROC-capable check out this video:

https://youtu.be/OBYUL5jOQKw

The same author posted about it in a forum capture I found, PDF below. VROC is not yet possible on the Z840 and likely never will be, but it is now available on the next HP generation (Z4 G4, Z6 G4, Z8 G4, for example). You're not asking about that, however.

For 4 NVMe sticks running at full speed on the Z840 you'll need to put the ZTD QP in the x16 electrical lanes slot that HP recommends and then set that slot's bifurcation in BIOS to x4x4x4x4. Leaving it at the default of "Auto" won't work. The 4 sticks will then show up as 4 separate drives in Device Manager.

Device addressing: HP documented for the ZTD G1/G2 and the ZTD QP an addressing mechanism so multiples of those won't interfere with each other. I'm working on figuring out the same for the ZTD DP (which is not as well documented) and will post on that later. A PDF is below that includes some important addressing info from HP for if you install more than one ZTD card of the 4 types. Below is a picture of the ZTD Dual Pro's E1 and E2 addressing 3-pin card headers that I need to create a decoder ring for. Note that the pin numbering is reversed for the second (E2) header, and this is how the card's jumpers came brand new from HP (both jumpers are actually shorting pins 2-3 if you look closely... the little white triangle printed on the PCB is always by pin 1). I'm guessing that the two addressing slider switches in the ZTD QP will correlate with the two 3-pin headers in the ZTD DP...

Top right corner of ZTD DP card

DGroves likely will weigh in on your questions. In the past he has commented on different non-HP NVMe M.2 SSD adapters and is quite experienced on this all.

[Fiendish]

Many thanks SDH, I had completely forgotten about the Turbo Drives, it sounds like a very sensible option and the numbers are impressive.  I do have the x16 slot 4 free so was going to use that anyway. It's very long card though, there'll be some cable rerouting for sure. 

So the addressing only comes into play if there are multiple ZTDs, is that correct? I only plan to get the one. 

[SDH]

I got my no-drives ZTD Dual Pro card straight from HP for a bit over 100 USD... I wanted to get the manual plus see what the defaults were. I'll PDF the manual shortly, but it has no info on LED activity cable placement or on card addressing. I don't really need LED activity to show up on the front of the workstation, but they included one of those LED activity cables in the unopened packaging. The card has a x8 electrical lane bottom PCIe slot interface. That on the ZTD Quad Pro is longer... x16 electrical lanes that can be split via bifurcation (quadrification?) to four x4 electrical lanes.

You don't really need to worry about addressing if you'll only be running one ZTD Quad Pro... once you set bifurcation correctly the drives will be seen by the BIOS and OS automatically. Read that PDF from HP with a few mods from me because HP does have some good tips in there. You could start with one stick and add more later. I've been finding nice very low hour 1TB sticks on eBay. With good searching skills I'm getting those at about 40-50 USD each. They don't need to match so if there is a particular one you want for your boot stick just put that in the primary slot and fill the other 3 slots as needed. You don't address the sticks... the OS does that if bifurcation is set properly.

A tip... from all I have gathered to set up a ZTD of any type as a boot drive you want to have zero added drives of any kind in your workstation beyond a single target M.2 stick during your clean install. You can add SATA drives or other M.2 sticks later but let the OS and BIOS settle in on that one boot M.2 stick before you add more. It will be less confusing to you and the installer that way.

[Fiendish]

Many thanks, some great tips there and I'll see what HP are selling themselves too.

I wasn't planning on using one as a boot but given the huge boost it would be over the Sata SSD I have currently maybe I should.  I was thinking about one or two Firecuda 530 4TB to start with for video editing use.

[SDH]

The Firecuda 530 is a PCIe Gen4 stick... it should be backwards compatible into your PCIe Gen3 Z840 no problem, but the Gen4 slots don't show up until the ZX G4 next generation of HP workstations. I'm at the experimentations stage, keeping costs down. I'm saving up for my plasma holographic cables...

[Fiendish]

Yes it's a Gen 4 which is of course is overkill, it just does seem to have excellent reliability and I'm going to be using these on paying jobs. But if you had any suggestions for any Gen 3 4TB sticks let me know! I think it's only the Crucials really

[DGroves]

if you are going to use the HP quad SSD card for video work then the selection of ssd's is quite critical.

you want/require ssd's that are ideally MLC 2bit nand based like the samsung 970 pro,......

3 bit "TLC" based drives are not recommended for professional use but may suffice for home use if they are high end PERFORMANCED based  and have a large SLC cache that is greater than the size of the video file you are using if you use a TLC ssd that has a 2GB SLC cache and your file is 3GB or larger then the ssd will have a major drop in speed causing video glitches 

most used data center ssd drives on eBay are still MLC based but you have to be careful, as some have firmware optimised for fast reading not writing check mfgr specs before buying and skip if data sheet on drive specs can not be found

the nice thing about datacenter ssd's are that the published specs are actually real world numbers that you can expect unlike the consumer specs that are mostly wishful thinking

if your video is 4k or less, (8k demands some cutting edge /expensive bits of kit)

you can also do quite well even today with spinning mech based "SAS" drives connected to a HIGH END PCI-E Raid card

such as the LSI 9361 or the newer 94xx series or the Adaptec ASR 8xxx line the adaptec cards while slower than the LSI cards are much more flexible with Raid and JBOD support with true passthrough which means the drive on the asr controller will bee seen on any other SAS HBA/Raid card

https://storage.microsemi.com/en-us/support/raid/sas_raid/asr-8805/

[Fiendish]

Many thanks, some great food for thought there. I have already filled all my 2.5" and 3.5"" bays hence the reason I was looking at an M.2 NVMe solution for further internal storage. This will be a stop gap until I get a more traditional external miniSAS raid with Areca / Adaptec / ATTO card. 

To this end I also want to maximise the storage I can get so I am looking at 4TB sticks. Gen 3 Samsung enterprise PM983 was MLC  and available in 3.84 TB but it is not now widely available. What do you reckon to the specs of the gen 4 PM9A3?

https://image.semiconductor.samsung.com/resources/data-sheet/samsung_ssd_pm9a3_data_sheet_rev1_0.pdf

The gen 3 can still be bought in the UK but it is roughly double the cost

[DGroves]

for video work the "U2" format is the better choice as the limited size of the M.2 format can lead to the SSD overheating unless used in a card with proper cooling  the the HP quad which has active cooling and is a workstation  type card is a good choice

the PM9A3 SSD appears to be a good choice for video editing based on the samsung website data sheet

Samsung PM9A3 vs. Samsung PM983

Differences between Samsung’s PM9A3 and PM983 (the company’s last-gen data center SSD) are significant. Besides the jump to the Gen4 interface, the PM9A3 offers better NAND and a new controller (V6 TLC and Elpis 8-channel, respectively) compared to the PM983’s V5 TLC NAND and Phoenix 8-channel controller. The PM9A3 also doubles the maximum capacity model to 15.36TB. And, as indicated above, the PM9A3 is available in a wide range of form factors, while the PM983 is only available in M.2 and U.2.

For performance, the PM9A3 (U.2) is quoted to deliver sequential reads and writes up to 6,900MB/s and 4,100MB/s, respectively, while random performance is expected to hit up to 1.1 million IOPS read and 200K writes. Samsung indicates that this is a 3.6x improvement over the PM983.

more drives can be inserted in the z840 by using 5.25 trays that can hold up to 4 or 6  2.5 in SAS or U.2 format SSD's 

https://www.amazon.com/ICY-DOCK-ToughArmor-MB608SP-B-Removable/dp/B07DM2NQNW/ref=asc_df_B07DM2NQNW/?...

https://www.amazon.com/ICY-DOCK-Backplane-Comparable-Tray-less/dp/B00V5JHOXQ/ref=asc_df_B00V5JHOXQ/?...

https://www.storagereview.com/review/samsung-pm9a3-ssd-review