Z620 booting from Intel and Samsung NVMe drives (with OPROMs)

[SDH]

I've been working on this project for some time, with help from DGroves and others over the years. Finally have made the breakthroughs. This is an advanced project. The three main Intel and Samsung drives I have done this with are NVMe-controller drives and also have added OPROM code built into their firmware that is donated from the drive over to supplement the HP BIOS boot code, allowing the boot of these ZX20 workstations to proceed. The HP ZX20 workstations do not fully include the necessary NVMe boot code in their BIOS without this supplement, but the next generation ZX40 workstation's BIOS does. The ZX20 era workstations do have a somewhat rudimentary UEFI capability which has improved over the next 2 generations (ZX40 and then the ZX G4). The supplemental code donated from the OPROM on these unique drives is what lets NVMe boot proceed in the ZX20 workstations.

Added breakthrough details to know is that the Samsung Magician 8.2 (EDIT: and the newer 8.3) free utility still includes a firmware update for the 950 Pro M.2 SSD, and that the latest Solidigm Storage Tool 2.1 free utility still includes the latest firmware and "bootloader" updates  for the Intel DC P3700 (plus all the lesser datacenter and client versions Intel released for their closely related PCIe3 x4 cards). You may know that Intel sold their SSD division to SKHynix, now branded as Solidigm, which has taken a more open-source approach. Those two software packages run well under both Windows 10 and Windows 11 as does a related software called Solidgm Synergy Toolkit. I show that in use below, which even allows setting added overprovisioning in the Intel DC P3700 above the 25GB Intel already included in its 800GB capacity version. That Toolkit also provides other valuable features. I use it for easy access to real-time temperature monitoring of the P3700 (and other vendor's drives) during my Samsung Magician peak performance testing. This P3700 started out at 32C, the performance test lasted for several minutes at 100% speeds, and temps elevated to only 35C. Clearly thermal throttling will not be an issue. Similar more-than-adequate cooling is present in the HP Z Turbo Drive G2 PCIe card (with its excellent aluminum heatsink), running the Samsung 950 Pro M.2 SSD at 100% performance. I'm testing these in PCIe slot 4 (PCIe3, x4 electrical lanes) of the Z420 v2 and Z620 v2 workstations. I expect same in the Z820 v2.

If you choose to get a P3700 make sure to not get an OEM version with OEM firmware on it. The Intel (->Solidigm) firmware updaters will not work on those. My original firmware (ended in "31") smoothly updated to the latest (8DV101H0) and the latest bootloader BIOS supplement (8B1B0133). A caution: For some of the eBay P3700 PCIe card drives I've seen very old firmware (listed on the top card edge label) as "TBD" = To Be Determined engineering samples. I'd stay away from those. Finally, there is a whole Intel series of lesser versions of this PCIe card, all of which use the same latest firmware and which can also be updated the same way with the Intel Command Line Interface DC software and the newer Solidigm Storage Technology utility/Synergy Toolkit. Those all run perfectly under W10Pro64 and W11Pro (always x64).

I'll soon be adding to this thread with some added HowTo information. Special appreciation again goes to DGroves... his generous tidbits over the years here added up to providing critical pieces of the puzzle. Several pictures for now, NVMe boot drives working very well in our Z620 v2 testbed:

MLC NAND, lots of extra from Intel, plus 10% overprovisioning set

Both use same techniques to work perfectly...

Speed testing on the P3700...

Sorry for the technicolor but you get the idea. Both are using the W11 Standard NVMe Controller driver when checked under the Storage Controller section of Device Manager. There is no need to try to dig out old drivers because W10 and W11 have them built in.

[SDH]

Below is some added info on this project, now working very well.  This was done mainly for the challenge.  I'll provide basic information that will let other forum members proceed and succeed with less effort, over this and the next post.

1. Certainly you should do GPT partitioning and a long-type NTFS format to prepare the drive. This eases use of the early incomplete UEFI features present in the ZX20 BIOS. The next ZX40 and subsequent ZX G4 workstations have BIOS with UEFI capabilities that are much more advanced. I use the Windows DiskPart and Disk Management built in utilities for that, with a DiskPart tip document attached below.

2. Choose the OPROM-supplemented drive. These include the Samsung 950 Pro M.2 form factor SSD and the Intel NVMe P3700 (or other in that line that included the OPROM).  The M.2 should be mounted in a Z Turbo Drive G2 PCIe card, and the Intel drives come as self contained PCIe cards. You want to use the PCIe3 slot 4 ideally but also can use the lower PCIe x16 slot. I have attached below a listing of the Intel PCIe cards that contain the OPROM code and note that they are highlighted and all have the same latest firmware updates. That also includes both the multiple Intel Data Center versions and the single Intel Client version. Soligigm (part of SKHynix) bought the Intel SSD division and included freely available firmware updates plus the tools to do that in the Solidigm Storage Tool. This was a very welcome change that greatly eased the update process. You can see what is the latest now and upgrade to that easily. Note that an OEM version of these drives will almost always have its own firmware which this tool will not upgrade.  See the 6/24 listing from Solidigm below. Here are 2 pics of before and after my DC P3700 was firmware updated:

Before...

After...

3. The Samsung 950 Pro firmware update can easily be done with the free Samsung Magician utility, most recent is 8.2.

4. Windows 10 and 11 have appropriate drivers for running these NVMe-enabled types of drives. The NVMe code in their OPROM gets injected into the boot process in the ZX20 workstations.  I have not needed to try to chase down old drivers from Intel... they are in the box already and the Windows Device Manager will detect them and provide the drivers automatically.

Coming shortly... the critical navigation in the ZX20 BIOS to enable the workstation to access the nascent OPROM NVMe

boot code, the use of Microsoft's MBR2GPT utility that allows you to convert a MBR boot drive over to a GPT boot drive so you don't necessarily need to do a clean install if you have a MBR partitioned M.2 HP AHCI-controller Z Turbo Drive already running your ZX20 workstation and want to shift up to. Finally, a few details on setting boot order both for the "full time" and the "one time" ways to do that.

[SDH · Accepted solution]

The info below should get you up and running. Attached is an Intel Product Brief for these Data Center NVMe drives with OPROM boot code in the drives that allow use in the ZX20 workstations if you know how to turn on that capability in the BIOS of these workstations. The same techniques work for getting the Samsung 950 Pro M.2 NVMe drives to work perfectly when using the HP Z Turbo Drive G2 PCIe card. The product brief shows two main form factors. These are (1) a 2.5" form factor NVMe device that may work in a U.2 (SFF-8639) adapter, and (2) a PCIe3 x4 half height card with a full height backplane plate. I've only worked with the DC P3700 half height card version. The 2.5" NVMe drive is something I've not personally seen used in HP workstations. The Samsung 950 Pro max capacity is 512GB but the P3700 goes up to 2 TB. MTBF is 2,000,000 hours... "230 years".

It is critical to clear the drive, GPT partition the drive, and then NTFS format it. If the drive is MBR partitioned it won't be seen properly by BIOS. Microsoft created a special utility for W10 and W11 that allows nondestructive conversion of an MBR-partitioned boot drive over to a GPT-partitioned boot drive. It works very well.

ZX20 BIOS Settings:

1. Use latest BIOS. Set to factory defaults and save on the way out of BIOS. Now check and change...

2. F10 into BIOS/ Storage tab/ Storage Options/ SATA Mode/ Default is RAID + AHCI (can change to just AHCI).

3. Now go over to the Advanced tab/ Device Options/ Mass Storage Option ROMS/ change from default of Legacy to EFI.

Save and back out of BIOS and save the changes as you exit BIOS.

Setting up the drive:

1. It is best to only have the P3700 in place, or the Samsung 950 Pro inside a HP Z Turbo Drive G2 with its nice big aluminum heatsink. Both are single slot width devices. HP recommendations for positioning of ZTD cards in the ZX20 workstations is in the PDF below, and I'd follow those if you are running a P3700 or a Samsung 950 Pro as your boot drive.

2. In HP BIOS there is the "1-time boot" section you get to via pressing F9. There also is the "full-time boot" section you get via pressing F10. I have found pressing at a rate of about 2/second is best and keep pressing at that rate until BIOS finally pops into a BIOS window. Then you have control. Also, I find the better F10 access is if you do that from full power off (a cold boot) and start the 2/second pressing of F10 just after you push the power on button.

When things are prepared correctly and you F10 to get into BIOS the OPROM-enabled 950 Pro or P3700 will be present for use in two sections... the upper UEFI and the lower Legacy sections.  You get to those after setting BIOS above, and then do a cold boot, and F10 as above/ Storage tab/ Boot Order/ UEFI Boot Sources/ In Hard Drive section move Windows Boot Manager up to the top and have USB hard drive below/ gray out USB Floppy/CD by disable with F5. Ignore the Legacy section below... the Z Turbo Drive likely will also show up there under something like PCIe slot 4.

If you have a ZTD G2 running a Samsung SM951 AHCI-controller drive and want to clone that over to a 950 Pro it can be done. I tune up the SM951-AHCI first fully including up to W11 24H2, capture its image on my USB3-attached external archive drive, swap in my ZTD G2 with its fully GPT/NTFS prepared 950 Pro, boot from the Acronis USB (plugged into the front top USB2 port on the ZX20). I use the 1-time F9 method to get that, chosen in the UEFI top section. I then clone the image onto the new drive having Acronis set to shut down when done. Remove the USB and the external HDD. On first boot chances are I'll get a blue screen. Try again, another blue screen but often the option comes up to try to repair in a WinPE environment. It is best to navigate to the end option of an elevated CMD recovery and there I run sfc /scannow. This repairs the issue and from that point on all is well. This only appears to happen if I clone from one type of drive over to another. It does not happen on clean installs. Once the error has been corrected an image capture/clone onto another same type drive works fine the first time. That trick is worth knowing... it saves losing an excellent install.

Finally, related. My main workstation that was running licensed software and complex imaging calibration was on a MBR-partitioned SM951-AHCI and I wanted to shift it to the GPT-partitioned P3700. I used the Microsoft MBR2GPT utility capable of converting a MBR-partition boot drive safely over to a GPT-partition. That was created to work under W10 and W11. I had first captured an original MBR -partitioned image for backup. I used that to clone a duplicate and then converted SM951-AHCI over to GPT. I finally cloned the GPT-partitioned image over onto the P3700, ran into the expected blue screen, got to the elevated CMD option, ran the sfc disk repair, and things have been running successfully thereafter. It was surely easier than doing a full clean install.

Interesting project. I think I have about an average of 225 years to go before my first P3700 boot drive failure.

[DGroves]

and a quick FYI, i use the intel dc3605 which is a 1.6TB SSD   using Sun/Oracle OEM firmware instead of the intel firmware

i've also tested the dc3800 models and they also work as bootable in the Zx20 systems

[SDH]

DGroves, thanks for that update. There seems to be quite a few of these NVMe-controller data center drives that had supplemental boot code included on board so that insufficient BIOS boot code on earlier servers and workstations could go ahead and boot with this new and very fast technology. The components used were of highest quality and with long projected lives. The ones I have gotten from eBay have had up to 97-100% life left, and some sellers even state the % life left in their ads. I've found good prices also on the Samsung 950 Pro with careful shopping. Your Intel/Oracle P3650 experience is interesting. Is there a Windows based firmware update method? So far, I only found a Linux method mentioned (which I am fully inexperienced with) and the latest appears to be Oracle FW RA13.

Below are my speed testing results using the latest Samsung Magician 8.2 running its standard Performance test, with temperatures recorded from baseline to max temp at end of the test. I used the Solidigm Synergy ToolKit for that. I'm seeing good temperatures maintained in all cases, with the HP Z Turbo Drive G2 excellent in its cooling capabilities for all M.2 sticks tested. The original Intel Add-In-Card design cools very well. As many of these were bought for servers many only have the 1/2 height metal backplane, but there are non-Intel full height replacements easily sourced from eBay that fit perfectly for low cost. Here's the speeds:

Excellent speeds and temperatures

Note the very high random read/random write performances for the P3700 in both the AIC and U.2 form factors, and also for the M.2 Samsung 950 Pro. Those are quite impressive. The M.2 950 Pro largest size produced was 512GB I believe, but the Intel P3700 (and others with the OPROM boot code onboard) is available in up to 1.6TB, maybe 2.0TB. Below are the four I tested with, the last 3 being NVMe-controller drives containing extra OPROM code to allow ZX20 boot from these. I used Acronis 2021 to clone the same Z620 v2 image onto each of these individual drives, so the hardware and software otherwise were identical:

This used the official fastest M.2 certified for the Z620

The Z Turbo Drive G2 had no problem keeping this HP 512GB SM951 AHCI-controller M.2 cool during stress testing. The Lenovo versions of this M.2 worked perfectly also. Up to this point these M.2 drives were the fastest drive option available for the HP ZX20 generation of workstations. No longer...

Another excellent M.2 NVMe-controller SSD with OPROM code included

These are excellent M.2 drives but somewhat hard to find for reasonable prices. I've found a couple of the 512GB versions to test with, lightly used, for between $50 and $75 USD. Samsung did not include the supplemental OPROM boot code on their later consumer M.2 Pro drives.

My favorite...

These add-in-card data center NVMe SSDs are very well built, cool very well, and were the easiest to get up and running. They have four rear LED indicators that can be handy for reference to (PDF attached below). The U.2 form factor drive shown below has none. For both of these P3700 SSD form factors, and for the 950 Pro, I'm simply using the Windows NVMe driver built into Windows10/11. Don't be concerned if you notice the NVMe driver date is 6/21/2006 in Device Manager. Microsoft updates the version number but not the date on purpose.

The P3700 U.2 SSD in StarTech U.2 interface card

This was an interesting part of the project. The 2.5" U.2 form factor can be interfaced with the PCIe bus using this PCIe3/PCIe4 x4 card. StarTech has released several versions over the years (all with the same part number) and this is an example of their latest (available now for a number of years). It has worked perfectly here. The 2-pin LED header shown is polarized + and -, and I hooked up a spare LED to it for grins (red wire +/black -). Works fine. The jumper shown down below the U.2 drive is worth having... it is included on the latest version of the card. My P3700 U.2 drive on first install did not show up in Device Manager or Disk Management (W11 24H2), so I just shifted the jumper from default of bridging pins 1-2 over to bridging pins 2-3. The drive could then be seen in each of those Windows utilities. I then used Diskpart to clean and GPT partition the drive and then did the long version of NTFS formatting via Drive Management. Shut down/shifted the jumper back to its default position, and things have worked perfectly thereafter. 

I also had a little issue with the Solidigm Storage Tool not seeing the original firmware as being updatable. I ran the older Intel SSD_DCT 3.0.27 Win64b (located via google) to use its command line interface (use an elevated CMD prompt) and entered isdct show -intelssd and that showed an update to the latest version did exist inside the tool. I then triggered the update with a CMD of isdct load -intelssd 0 to do the update. The latest firmware is mature, ends with 0H1, and has been that for years now (since 2017). The Solidigm Synergy ToolKit also has firmware probe and update capability. Not sure if the issue was a temporary glitch or if the Solidigm Storage Tool was simply missing that FW as an oversight. Those are two different Soligigm programs. The U.2 drive now works just as well as the Intel add-in-card version.

I hope this helps... it will be best to do a clean install instead of my clone approach, but even multiple clones worked well here. The Z620 v2 I worked with has some very complex software and licensing on it and I had no interest in starting from scratch. 

Again... kudos to DGroves for all his help on this over multiple prior posts here, over the years.

[DGroves]

yes oracle used linux for firmware updates to the 3605 

servethehome appears to still have a copy of the 3605 RA11-13 firmware available

https://forums.servethehome.com/index.php?threads/1-6tb-intel-dc-p3605-p3600-800.9973/page-2

and yes RA13 still seems to be the latest oracle for the intel based  3605

if your 3605 has RA10 or lower, you must upgrade to RA11 before upgrading to RA13

for RA11 or RA12, you can upgrade directly to RA13

you can use a live ubuntu OS  to flash

http://www.oracle.com/us/products/servers/f160datasheet-2900793.pdf

sudo add-apt-repository universe

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install nvme-cli
sudo nvme list
sudo nvme fw-download /dev/nvme__ --fw='/__/8DV1RA11.bin'
sudo nvme fw-activate /dev/nvme__ --action=1
sudo nvme reset /dev/nvme__
sudo nvme list
sudo nvme fw-download /dev/nvme__ --fw='/__/8DV1RA13.bin'
sudo nvme fw-activate /dev/nvme__ --action=1
sudo nvme reset /dev/nvme__

[G-Windows1]

Hi SDH,

Got an email notice of your PM, Subject "Test."

I lost my access to my G-Windoze account, and am now G-Windoze1.

I can't seem to find the "PM" button anywhere, so I am unable to send a PM at present. But wanted to let you know of my new screen name. Anything sent to G-Windoze is out of my reach!

[SDH]

Well, the "test" message was sent to your old name plus one other name that was somewhat like yours. That somehow made it. When I tried to send to your new name the PM system seemed to think you don't exist. See attached pic.

The only thing I was going to send was pics of the 32GB and 64GB memory sticks from HP for the Zx G5 workstations. If that is not needed lets bag it and have you stick with 16GB sticks.

[G-Windows1]

SDH,

That bubble error message is puzzling, and baffling. Perhaps I messed up my new screen account by using the same email as for my original, G-Windoze. I may abandon this account, also, and create a new one using a different email. But I should be able to use something like G-Windoze2 or 3 for my screen name.

I'm planning on getting 128GB total. I am willing to either 1) buy 7 more 16GB sticks (or 8 if necessary); 2) pull the 16GB stick and use 4x32GB; or 3) pull the 16GB stick and use 2x64GB. Not sure which configuration is most efficient. Minor cost differences among the options is not of concern.

Sorry that I'm off-topic on this thread, but HP has tied my hands!

[Ivo_555]

Has anyone managed to install and boot Windows from an Intel DC P3608 SSD on HP Z620?