HP Z440 Workstation, Kingston NV2 NVME connected via ZTD G2 is not detected after reboot.

[lp1641]

It's going to be a long story. I went through all the forums, all the videos and all the comments to find some solution and found nothing that would help me.

I have a HP Z440 workstation, 3 Hynix 16gb RAM and 1 Kingston 16gb in quad channel mode, Xeon 2690v4 processor, Samsung 1TB 870 EVO SSD, Z Turbo Drive G2 card-adapter from HP through which I connect Kinston NV2 SSD  512gb gigabytes, dvd drive, and finally video card GTX 1050Ti, the rest of the peripherals in the form of mice and keyboards to list probably not worth it. And of course Windows 11 operating system on a GPT partition table boots in UEFI mode. I have BIOS version 2.61, the latest at the moment and I've tried various resets already.

The Z440 has the following PCI slots in that order.
SLOT 1 - x1, 2.0
SLOT 2 - x16, 3.0
SLOT 3 - x4, 2.0
SLOT 4 - x8, 3.0
SLOT 5 - x16, 3.0
SLOT 6 - PCI

HP in the manual for the Z Trubo Drive writes about plugging into slots 3, 4, 5.

My goal is to put the video card in SLOT 2 and the ZTD with my Kingston NV2 in SLOT 5 because that way it provides the greatest distance between the video card and other components and better cooling, but I can't do it. I've already tried all the slots and I'll tell you how it works with each one.

SLOT 1 - The system boots, also the system boots after restarting the computer, the system is slow, Crystal Disk Mark shows 400mb/sec and the computer starts to freeze, in general a bad slot to run such an SSD.

SLOT 2 - The system boots only if you unplug the computer and wait and then turn on and not always, if you reboot from Windows at the next boot NVME will disappear from BIOS and the system will not boot, will write that there is nothing to boot from. Speed 3000+mb/sec.

SLOT 3 - System boots, system boots after restarting the computer, speed 1500mb/sec. The speed is lower than that of 3.0 it is clear, the only disadvantage is that after reboots NVME disk drops to the lowest position in the boot priorities, which means that if I have inserted a stick with Windows installer, for example, it will choose it instead of NVME because that for some reason was rediscovered and was at the bottom of the list. So this is solved by disabling all possible boot options except NVME, if you have several OS then you're out of luck I guess. In general the most normal variant but with this variant the video card becomes in SLOT 5 and is too close to the bottom of the case, bad cooling, high temperatures, in general I am sad about it.

SLOT 4  - NVME is sometimes detected after unplugging the computer, but mostly it is not detected at all, if it is detected, it disappears after reboot. Speed 3000+mb/sec.

SLOT 5 - I'd like to use it in this particular slot. The BIOS has never seen the drive in this slot no matter how many times I change the settings, unplug the computer, or reset the settings. It just doesn't see NVME and that's it.

I tried to change all settings related to SATA controllers, with Secure Boot, with Power Management and many others, with delays for PCI cards in Boot Options, as well as with Fast Boot and other stuff, tried to change the speed of PCI slots and bifurcation, tried to disconnect all unnecessary slots, all other disks and all USB devices. None of this affected the behavior of PCIe NVME in the slots that I described above. Just nothing helps and that's all.
I also tried another adapter "Maiwo M.2 NVMe M-key SSD to PCI-E 3.0 16x / 8x / 4x (KT046)", there are a lot of these sold nowadays, and it works exactly the same as the Z Turbo Drive G2 card from HP, wherever I put the SSD the behavior is the same as I described above per slot.

That leaves me with only two options. Another SSD and I have a defective motherboard.

I have only two options left. The problem can be with the SSD and I need another one, I don't have another one to try yet, and buying one for the sake of such an attempt is certainly not pleasant or I have a defective motherboard and then all my efforts were in vain from the beginning.
Strangely, the video card works in both x16 slots as it should, so the slots themselves probably work.
Also everything else is working properly as well.
The general theory is that the PCIe adapter works fine, almost fine, only in the 2.0 slots. That's my experience. I'm thinking of picking up a Samsung 970 EVO Plus and making one last attempt.
GTX 1050Ti is not a powerful video card, but even it heats up to 80 degrees when it is in SLOT 5 and close to the bottom of the case, so this position is not suitable for it, and I was planning to do a video card upgrade, and this will be a sad situation, so I want the video card to be in SLOT 2 and the disk in SLOT 5.

I don't think it's Windows because SLOT 3 gives you the ability to install anything and then it always sees the NVME disk even if there is nothing on it, so if the BIOS doesn't see the disk you won't install anything there, it seems logical.

I saw a comment by someone on youtube who had many of these workstations and he said that half of them lost disks after rebooting, so this doesn't give me much hope of solving the problem, maybe it really is a factory defect.

If the video card is installed in SLOT 2, it covers SLOT 3 and goes almost next to SLOT 4, so they remain unavailable. Because of this, I have to put it in SLOT 5 to get my NVME SSD to work.

[SDH]

DGroves has been an excellent source of information on the ZTD issues, and he has helped me through getting W11 up and running on the Z440. HP's documentation of the ZTD details is a bit confusing at times. My advice is to get back to basics.

1. Clean OS Install with only the ZTD in place and no other SSD or documents HDD present. You can add those later. The ZTD will be your boot drive.

2. Set BIOS to factory defaults before doing a clean install. That can be done from within BIOS but you even might have partially corrupted BIOS given your symptoms so doing the long type of CMOS clear/reset is what I'd do at this stage of your issues. That has cleared up weirdness for a number of us, and that also will get your BIOS to factory defaults.

I have found that doing upgrades from a pre-existing W10 build to W11 results in some old drivers being dragged along into the new build. Clean installs worked; upgrades did not in the Z440. Upgrades from W10 to W11 did work for the Zx20 generation of workstations but some old drivers were dragged along into those builds so I concluded that we need to only do clean installs on those too.

3. Format the ZTD using long type NTFS method (not "Quick"), with MBR partitioning. In the Power Options app in Control Panel for W10/W11 we always turn off Fast Startup.

4. Run BIOS in Legacy mode, not UEFI. DGroves explained that UEFI in the Zx40 workstations is subpar (but good in later generations of HP workstations).

5. Slot positioning: Use slot 4 which is a PCIe Gen3 8 electrical lanes slot. HP provides the "order of preference" for the Z440 (and Z640) in placement of the ZTD card: First is slot 4, second is slot 5, third is slot 3. However, slot 3 is a PCIe Gen2 slot instead of PCIe Gen3 and thus will reduce read/write speeds to about 1/2 of a NVMe M.2 SSD's potential. Your video card should go in slot 2.

Of interest with the newer Z440/Z640 versions of BIOS you can turn on bifurcation for slot 4 and 5 (to x4x4 / x4x4x4x4, respectively) allowing you to use a ZTD Dual Pro with two NVMe M.2 SSDs in it should you choose to do that. If using a ZTD G1 or ZTD G2 card just leave that BIOS setting at default.

6. Pay attention to cooling... add an official front case cooling fan if you don't have one yet. We also use the memory cooling "saddle" that provides active cooling to the Z440 front and rear memory banks. Those are still easy to find on eBay and inexpensive. I can get you the part numbers if you need.

6. Speaking of heat, we've found we don't need to worry about that with the ZTD G2... it has the nice big-surface-area CNC milled aluminum heat sink. I've never been able to force thermal throttling of a NVMe stick in that card, and I've tried.

7. You can find used smaller capacity (say 256GB or even 512GB) NVMe M.2 SSD sticks on eBay quite easily and I've used both HP and Lenovo OEM ones in the ZTD G2 to experiment with. You can even just use a ZTD G1 to mount those sticks in (those come without a heatsink) for experimenting with. That allows you to dink around more easily. Both NVMe- and AHCI-controller M.2 sticks will work fine in either the ZTD G1 or ZTD G2 cards, and I've run both types in a Z440 simultaneously using 2 ZTD cards (in slot 4 plus slot 5). If you use two ZTD cards simultaneously you need to remember to shift any one of the 3 green addressing jumpers leftward from its default far right position.

Hopefully this will help get you going. We're having none of your issues here.

[lp1641]

Thanks for the reply, the actions seem pretty generic, not sure if this can help.
I did another interesting test in the meantime, I have my previous computer DELL T3500 there is no support for nvme drives at all, I decided to try using the adapter I mentioned earlier "Maiwo M.2 NVMe M-key" because the HP was unlikely to be usable.
And what I found out is that the DELL T3500 has all slots working with PCIe 2.0, I was not able to boot from Windows 11 which was installed on the SSD using the Clover flash drive I put in the DELL T3500, but I was able to open this NVME as a file drive. On my DELL T3500 I have a SATA SSD with Windows 11 installed on it, and I was able to open partitions from Kingston NV2.
So here are the results of my testing on the DELL T3500.
SLOT 1 x4 - disk is available and after rebooting too.
SLOT 2 x16 - disk is available or not available, usually available if you unplug the computer and then turn it on. The situation is similar to SLOT 2 on HP Z440, which is interesting.
SLOT 3 x4 - similar to SLOT 1, works fine and after reboot too, the disk is available for reading and writing.
SLOT 4 x16 - NVME SSD is not detected. Similar to SLOT 5 in HP Z440, failed to make it visible.

So the new theory now is that Kingston NV2 does not work correctly in any slots that are larger than x4.

Very interesting that the slots in the DELL T3500 and HP Z440 where the SSD doesn't work at all are the same, they are the lower x16 slots, no idea where this is coming from why this is how this could be).

I have a Samsung 970 EVO Plus on the way, so it will be fun if I get the same results on it. But testing on the DELL suggests that the SSD drive itself may be the cause of the problem. That will be clear tomorrow.
I don't think switching to Legacy mode can be helpful because the disk is either visible in the BIOS or it's not, regardless of the mode. But I will certainly try it if the situation does not change with Samsung.
About formatting the disk completely to 0, that's interesting, maybe I'll try that too.
But about the long CMOS reset I don't understand what you mean, because I've already reset the settings several times, once through the BIOS itself, another time through a button on the motherboard that had to be held for a while, and then there was a case of rolling back to 2.59 then went back to 2.61 when I realized that it didn't help. So if you clarify what is "long reset CMOS" I would be grateful, and how it differs from another way to reset the settings.

DGroves  - Who's that? Is it the name of an internet resource or a username? Because I don't know 🙂 Can you give more information please?)

[SDH]

Long version of reset CMOS is more than just pushing the yellow button on the motherboard. There are motherboard capacitors that hold charge and can prevent full clear of partially corrupted CMOS or BIOS settings. Some attached peripherals (such as monitors) can feed back power to the motherboard (via their video cables). Computer attached to mains get power to motherboard even with computer off. CMOS battery needs to be removed during the process. All that... plus time and repetition a few times to drain the capacitors. Some ideas:  https://www.maketecheasier.com/how-to-reset-cmos-and-why/ 

DGroves: a valued member of this forum who is a high level IT professional with much experience working with HP workstations and servers and much other related gear. He has large-enterprise level contact with HP, and is very generous with his talents here. His posts should be easy to find.

I agree with your idea re Kingston... HP and Lenovo have been using mainly the Samsung OEM M.2 sticks. If you look on eBay for PM981 or PM981a with a HP or Lenovo added and filter for price and shipping lowest you'll find good deals. Turns out laptops coming with 1TB M.2 sticks sometimes get near-new upsizing of their original stick.

It turns out that Lenovo has a more active firmware update process and a tool for autodetect of the Lenovo stick in place and then upgrade if needed. That only works on Lenovo OEM sticks due to a little toggle in the firmware itself. Easy to find via google for lenovo nvme firmware update utility.

FYI I'm running one Z440 via a ZTD Dual Pro with a Samsung 980 PRO as boot M.2 above a Lenovo 1TB stick in the DP's lower M.2 port, with bifurcation turned on, PCIe slot 4. Very fast boot/documents drive combo. No HDD in that Z440. You don't get into PCIe Gen4 until the Zx G5 generation of HP workstations, but the 980 PRO is backwards compatible with Gen3.

[lp1641 · Accepted solution]

The situation has cleared up, today I received a Samsung 970 Evo Plus and it started without any tambourine and settings in the fifth x16 slot as I wanted, the video card on top and the disk in the adapter under it at a distance, the most optimal option and I am immensely glad that I was so lucky because I did not believe in success and this Kingston took me a lot of nerves while I tried to start it. I installed Windows 11 on a new disk, with Secure Boot in UEFI mode, everything works. Windows 11 installer loaded from a flash drive in UEFI mode immediately saw the new empty SSD, besides it was visible in Legacy mode as SLOT5 NVMe.
So if it works, it works and no special settings and tambourine dances are needed. At reboot it sees everything and the position in Boot Menu is saved. It is not clear only on what pattern some disks start and others do not, which makes it difficult to give advice to someone, but that Kingston NV2 here do not need to take it for sure.