HP Z440, Turbo Quad, NVMe performance issue

[GKre]

i have an Z440 expanded with an HP 841969-001 Z Turbo Quad Pro N2M99AA and four HP EX950 - SSD - 1TB - intern - M.2 2280 - PCI Express 3.0 x4 (NVMe) (5MS23AA#ABB) installed.

The first one is used as system. The other as regular disks with one partition.

Using AS SSD Benchmark, the first disk is bringing good values. But the three other drives are much poorer in performance.

This can be reproduced over rebooting the pc.

I wonder, why this is happening. any idea?

[BambiBoomZ]

GKre,

Without detailed knowledge of the configuration of the Z Turbo Quad Pro N2M99AA, a first thought is to consider that each drive for full performance needs 4X PCIe 3.0 lanes. So, four drives on one PCIe card would need to be mounted on a PCIe x16 slot- for example the second GPU slot.

I use a Samsung 970 EVO M.2 NVMe 500GB for data in a z620 (@ 4.7GHz) and that is mounted in the second GPU slot- it was the only PCIe 3.0 slot remaining- and the benchmark is somewhat better than the published specification:

This is only a guess. I hope others will comment.

BambiBoomZ

HP z620_2 (2017) (R9) > Xeon E5-1650 v2 (6C@ 4.7GHz) / z420 Liquid Cooling / 64GB (HP/Samsung 8X 8GB DDR3-1866 ECC registered) /GTX 1070 Ti 8GB / Samsung SM951 M.2 512GB AHCI + Samsung 970 EVO M.2 NVMe 500GB + HP/HGST Enterprise 6TB / Focusrite Scarlett 2i4 sound interface + 2X Mackie MR824 / 825W PSU / Windows 7 Prof.’l 64-bit (HP OEM) > 2X Dell Ultrasharp U2715H (2560 X 1440)
[ Passmark Rating = 6485 / CPU rating = 16583 / 2D = 854 / 3D= 13835 / Mem = 3159 / Disk = 14735 / Single Thread Mark = 2606 [8.7.21]

HP z420_3: (2015) (R12) Xeon E5-1650 v2 (6C@ 4.6GHz) / z420 Liquid cooling / 64GB (HP/Samsung 8X 8GB DDR3-1866 ECC registered) / NVIDIA GeForce GTX 1060 6GB/ Samsung SM951 M.2 256GB AHCI + Samsung 860 EVO 500GB + HGST 7K6000 4TB / ASUS Essence STX + Logitech z2300 2.1 / 600W PSU > Windows 7 Professional 64-bit (HP OEM ) > Samsung 40" 4K
[Passmark System Rating: = 6186 / CPU = 15845 / 2D = 819 / 3D = 11216 / Mem = 3047 Disk = 13905 /Single Thread Mark = 2525 [7.3.21]

[GKre · Accepted solution]

Thx @BambiBoomZ,

i did some more testing and the cause was simple but also stupid.

On those three devices the "write cache" in the settings was disabled.

Well - i wonder why this is affecting the performance so dramatically, even on reading data, but that seemed to be the cause.

And yes - the turbo quad is installed in a 16x slot with "bifurcation" enabled.

After changing the setting all devices show the same performance.

[DGroves]

and this is a perfect example of why workstations and datacenter/servers use SSD's that are MLC nand in most cases where read/write performance needs to be fast , also these ssd's published speeds are ones that the device actually will reach and maintain not the consumer ssd's market speak of "up to" that realistically can not be reached much less maintained

most consumer SSD's use a portion of the ssd to create a pseudo SLC section that buffers the data until the ssd can write it out to the Main memory of the slower TLC nand, and if you exceed this SLC buffer the SSD"s performance will dramatically drop, also MLC/SLC nand can be written to many more times than TLC/QLC nand

MLC= multi level nand (two state)

TLC = three level nand

SLC= single level nand

QLC= quad, or four level nand

so if you use your system to make a living, a consumer ssd most likely is not what you want, those users who game or just play around on the system might find that the tradeoff is acceptable

https://www.atpinc.com/jp/blog/what-is-SLC-cache-difference-between-Dynamic-Static-SLC-cache

https://ind.adata.com/en/technology/89

[SDH]

To GKre:

"On those three devices the "write cache" in the settings was disabled."

Could you please explain how you got to that setting, and how you changed it from disabled to enabled?  

Thanks....

[SDH]

Found it.  This is not a toggle option for the SSD firmware internally.  It is a system setting for each installed drive, and the default for an internal drive is enabled.  An external SSD might have this option disabled if it does not have power backup.

This feature has to do with maximizing speed of the SSD in a safe environment where there is power backup so the write-caching (very rapid events automatically performed if write-caching is turned on) has time to transfer data over from the volatile to non-volatile SSD memory in the event of a power loss.  A desktop or workstation with attached UPS power backup has this, as does a laptop with battery in place, and some SSDs have internal capacitors that provide enough power backup internally.  Instant loss of power to a SSD can fail its most recent data write if write-caching is turned on.  So, a super cautious owner without UPS on a workstation might turn that feature off for data safety.  However, the price is slower performance (as the OP's two images above show).  That is a bigger hit than I would have expected... less than half the total score if it is turned off vs on, and impacting both read and write scores (especially that write score).

However, I believe there are some pretty hefty capacitors built into HP workstation power supplies that also would provide power "backup" for the brief time needed even if a UPS was not being used.  DGroves, is there enough time provided by those, do you think?

To access the setting, and there probably are several pathways:

1.  Right-click on Computer, select Properties, click Device Manager on the left and expand the Disk drives section there.
2.  Select your SSD by right-click it, select Properties and go to the Policies tab. Make sure "Enable write caching" for the device is ticked, and hit OK.

Here is what the last step looks like:

[DGroves]

most enterprise ssd's have caps onboard to supply power until all data can be safely written to the ssd's onboard nand chip

almost all consumer based ssd's lack this feature so if power fails they most likely would corrupt data if you were in the middle of a large data xfer (like copying a 10 gig database file)

https://www.kingston.com/unitedstates/us/solutions/servers-data-centers/ssd-power-loss-protection

https://www.micron.com/-/media/client/global/documents/products/white-paper/ssd_power_loss_protectio...

i don't know if any workstation can/will provide sufficient power reserve to allow  ssd(s) without end to end power protection to always safely write out the data in it's ram buffer to the nand chip

also it's not just power available, that power has to be clean steady power and the systems power supply may not provide that just from it's filtering caps depending on it's hardware configuration

i always run my home systems from full sized ups's that can supply 15-30 min backup power reserves not just for this reason but the area where i live gets electrical storms in the summer (which just took out my eaton 1500 watt unit!)

the ups cost about 600 dollars to replace, but that was much cheaper than replacing all the equipment attached to it

i'm currently using two HPE branded 1500 watt eaton units that i bought off ebay (new old stock) w/batt for 130.00 each

the HP OEM uses the same hardware as my failed unit but HPE only enables some of the units abilities, still for 130 dollars the price was definitely right for me!

[GKre]

@SDH - you found the right setting.

What is strange for me, this setting seems to affect read performance also.

The Turbo Quad seems to have a large group of capacitors available, so i guess they might be helpfull in case of power loss.

Well - after changing this the SSD's all four came to the expected performance.

Thx to all for assisting...

[SDH]

GKre,

Yes, I also was quite impressed with how low the performance went with both Read and Write scores when the settings were right versus wrong.  Read 1134 -> 887, and Write 1090-> 59!  Total score cut more than half from 2788->1284.

I'd like to do this experiment here, but am a bit confused on your hardware configuration.  Did you have 4 of the same m.2 SSDs all in the one Z Turbo Drive Quad Pro?  Were they what came with the ZTD Q4 or did you load the 4 slots on that Q4 with 4 HP EX950-SSD-1TB?  Had you bought a "bare" 4-slot ZTD Q4?

Or, did you run a stock off the shelf ZTD Q4 and then add in four HP EX950 SSD 1TB elsewhere?  If so I can't figure out where you put them.

You clearly are onto something interesting here...

'

Thanks.... 

[GKre]

The setup is not very complicated.

I bought one Turbo Quad Card and four identical 1TB NVMe's from HP.

The card is in the slot mentioned in the manual for the Z440 (Slot 5).

There's only one more card (graphics card) in Slot 1.

The performance issue is reproducable. Also - I am not able to build a Storage Space out of three, empty SSD's.

The last, strange thing is, that the devices are recognized as removable devices.

Well for me it is not a big problem. i have the drives configured as Stand alone Drives and that works fine.