Booting from an NVME SSD card on a Z620

[Buffalo_Rider]

I will be posting on this thread the steps I took in getting my z620 to boot from a Samsung 970 EVO Plus (NVMe SSD) card.  There are some excellent posts in these community pages and some terrific people who helped me get this working - especially DGroves and campdublanc.  My many thanks to them.   My z620 is running great.

There is an archived post on the z420, z620, and z820 that has an older procedure in it that worked for some people.

There is a more recent post on the z820 and z600 which is where I found the pieces for getting mine to work.

I go into quite a bit of detail in the steps I wrote up.  A lot of you are a lot smarter than I am so could probably do with less detail.  However, by typing up the details, it helped me think it through and keep track of where I was.  I also think the detail will be useful for others.  So this is what I will post over the next few days.

[Buffalo_Rider]

Here is a summary of the results I am seeing.  I have had this up running only a few days, but I assume there won't be much deviation from this over time.   My NVMe card is a Samsung 970 EVO Plus, 500GB.  I am running Windows 10 Pro.

Performance: 

Performance on 2/10 (in parentheses is the first time I ran this a few days earlier) - using the Samsung Magician, its Performance Benchmark test.

Sequential Read:   3537 MB/s    (3514)                                 Random Read: 112,548  IOPS   (113,769)

Sequential Write:  3264 MB/s    (3274)                                  Random Write:  75,195  IOPS   (75,927)

Here, below, are some updated performance numbers after I installed the Samsung Driver for NVMe cards - on 2/13.   My 970 EVO Plus was running off a Microsoft (generic?) driver before (i.e.,  in the above numbers). 

Sequential Read:   3531 MB/s                 Random Read: 117,675  IOPS  

Sequential Write:  3274 MB/s                  Random Write:  97,529  IOPS  

The Samsung Driver improved performance for the random operations.  The temperature performance of the card did not change; the data below remains accurate.  

Temperature of the Card:

From Samsung Magician.  I have a passive (non-fan) heatsink on the card.  

Typical while doing some work is about 33 degrees C.

Typical while computer is idling, no programs up, is about 27 degrees C.

After running the performance benchmark:  41 degrees C

Highest I have seen (for only a few seconds, after a long write to the card):  46 degrees C.

If I can figure out how, I may add a screen shot here later. 

[Buffalo_Rider]

Step 1: Buy or obtain the parts you will need.

• The NVMe Card with an M.2 connector.
• A PCIe Adapter card– X4, X8, or X16 – with an M.2 connector, on which to mount the NVMe card. This is needed because the Z620 does not have an M.2 Connector on its motherboard.  I got an X16 card.
• An empty USB 2.0 Flash Drive of 32 GB or less. A mini-USB Flash Drive, such as a Cruzer Fit, is a nice size.  I got a 16GB USB 2.0 Cruzer Fit. (For non-US folks: USB Flash Drive = Clé USB = USB Key)
• Another empty USB Flash Drive, (either USB 2.0 or USB 3.0) but of 16GB or more to hold the Windows 10 OS for installation purposes.  I got a 32GB USB 3.0 Flash Drive. 
• (Optional, but a good idea): Either (a) an USB Internal Header or (b) an USB 2.0 to motherboard cable. The USB 2.0 Flash Drive (of 32GB or less) will plug onto the header (or cable) and the header (or cable) will plug into the motherboard’s USB 2.0 connector.   I got a USB 2.0 Internal Header (single-layer) similar to the ADPU2DU-NO3.  If someone prefers the cable options, this part should work: Tripp Lite P/N: U024-06N-IDC.
• Duet/REFIND code. (Get via a download. See instructions below)
• Windows 10 64-bit Operating System (Get as a download from Microsoft.com)
• (Optional, but a good idea): A heatsink for the NVMe card.  Keeping the NVMe at cooler temperatures will extend the life of the NVMe card, help prevent it dying, such as dying prematurely, or dying suddenly.  There are various heatsink options out there, most passive, but some with a fan.  I got a passive heat sink (the Warship) and it is working well for me - my NVMe card stays in a very comfortable temperature range.  I am using the NVMe for the OS and for programs.  I have all my data files on other disk drives to reduce the risk of data-loss should my OS disk fail - something I have been doing for years.  Cross my fingers, but so far I have never lost a file.

[Buffalo_Rider]

Note: The following three steps – Steps 2, 3, and 4 – can be done in any order.

Step 2: Mount the NVMe SSD Drive (Card) onto the Adapter

1. If you got a heatsink for the NVMe SSD Drive (Card), mount the heatsink onto the card.
2. Plug the NVMe card with the heatsink into the M.2 connector on the Adapter Card.
3. Install the mounting screw to secure the Card to the Adapter Card.
4. Set the Adapter-NVMe card aside for now, to install later into the computer. DO NOT INSTALL the NVMe Card into the Z620 PCIe slot at this time; my experience is that the computer will not power up.

Step 3: Install the REFIND code onto the USB 2.0 Flash Drive (of 32 GB or less)

1. Download a copy of the Duet/REFIND code.   DGroves provides a link in one of his replies on the post “Booting to NVMe ON Z820 HP AND Z600,” Message 41.  This is the code that augments the BIOS, that will allow the Z620 to ‘see’ the NVMe card, load Windows 10 onto it, and allow it to be the boot drive. The name of the download was “Duet_UDK2019_REFIND.zip”
2. Plug the empty USB 2.0 Flash Drive (of 32GB or less) into a USB port.
3. Run the REFIND code to create the REFIND USB Drive – the following is copied from a the post by campdublanc
   1. Connect a USB stick and go to the UDK2019 folder and start the "USBSetup" application
   2. Select your USB stick letter from the drop-down combo box. Warning: The following step will completely remove all information and partitions on the selected USB flash drive.
   3. Remove all information and partitions from the usb drive by clicking on: CLEAN.
   4. Optional, format the selected usb drive by clicking on: FORMAT.
   5. Copy the graphical boot manager REFIND onto the USB stick by clicking on: REFIND.
   6. Once done remove the Refind USB stick.
4.   Set the REFIND USB Drive aside for now.

Step 4: Create a Windows 10 Installation USB Flash Drive

1. Plug the empty USB Flash Drive (USB 2.0 or USB 3.0) that is 16GB or larger into a USB port.
2. Power up the computer you choose to use.
3. Go into Administrative mode on your computer, if you are not already logged on as an Administrator.
4. Bring up an Internet browser.
5. You can use RUFUS or the Microsoft Download Tool to create the Windows 10 USB Flash Drive.  

With the Microsoft Tool:

1. While in the browser, download the Windows Installation Tool from the Microsoft website.   It can be found by doing a search on “Windows 10 download” in Bing or Google.   It can also be found by typing in the URL, which is www.microsoft.com/en-us/software-download/windows10.
2. Go find, on your computer, the Windows Installation Tool you just downloaded. The one I downloaded in January 2020 is called “MediaCreationTool1909.”
3. While logged on as an Administrator (or more specifically, a user with Administrative privileges), Click on the Windows Installation Tool to bring it up.
4. Go through the screens. When asked if you want to “Upgrade this PC Now” or “Create installation media”, select “Create Installation Media” (then click on Next).
5. On the ‘Select Language’ Screen, remove the check from the small box to the left of “Use the recommended options for this PC.”
6. On the next screen choose “USB Flash Drive” (and click on Next). DGroves (others?) have said the NVMe Windows 10 installation must be done from a Flash Drive; said Windows 10 won’t install properly from a DVD.  {Update: Some DVD installations may work - see DGroves message below.}
7. If you have more than one USB storage device connected to the computer, you will likely get another screen asking you to choose the USB storage device on which to put the Operating System. Select your USB Flash Drive – and be careful to pick the right storage device – and then click on Next.
8. The program will write Windows 10 to the USB Flash Drive, formatted for installing onto your chosen storage device (the NVMe card, in this case).
9. Note that Microsoft’s "MediaCreationTool1909” does not ask anything about a GUID Partition Table (GPT), which is associated with UEFI, or a Master Boot Record (MBR), which is associated with Legacy mode. The Installation Operating System for Windows 10 that the tool writes to the Flash Drive accommodates either UEFI or Legacy, but the installation default is UEFI mode.
10. When the MediaCreationTool1909 is done, i.e., “Your USB flash drive is ready”, click “Finish”, give the program time to clean up the setup. The Win10 USB Flash Drive can then be removed.
11. If you are concerned about having a Product Key during the installation, there are programs on the Internet that you can download onto your computer. Run the Product Key program and it will provide you the product key for your computer.
12. Set aside the Win10 USB Drive (for now).

[Buffalo_Rider]

Step 5: Change the BIOS settings in preparation for the NVMe card

            The following steps 2-7, are copied from (and then modified) from the post of campdublanc, his steps 1-6. He was working on a z820. The BIOS of my z620 is different, so gives slightly different options. DGroves, with his excellent knowledge of BIOS, greatly helped me understand what to do.  Note: NIC means Network Interface Card

1. Power up the computer and tap the F10 key until the Setup Utility comes up. The Setup Utility is the tool for managing the BIOS settings. (Note: On the Z620, powering up and tapping the F9 key will bring the user to the Boot Setup Menu.)
2. Set the BIOS to factory defaults: “File->Default Setup->Restore Factory Settings”. Save the Defaults, then exit the Setup Utility.  The computer should then complete powering up to the Windows Desktop.
3. When the z620 gets to the Desktop, power the computer down.
4. Power up the computer and tap the F10 key until once again the Setup Utility comes up.
5. In BIOS disable AMT in two places: “Advanced->Management Operations->AMT” and in “Advanced->Device Options->NIC(AMT)Option ROM Download.” There is also an “NIC Controller (AMT) under the Security menu – this can be left as is; setting the two items above is sufficient.
6. In BIOS disable boot from network: Security->Network Boot.
7. Per campdublanc: ‘If not booting from the onboard LSI SAS/SATA controller, disable the LSI controller boot rom (do not disable the LSI controller unless you are using an add-in Raid card).’ This does not apply to my z620, nor the standard z620; it will apply if an LSI card (option kit) has been installed on the z620. So bottom line for me: skip this step.
8. Per campdublanc: ‘In BIOS disable the boot from network card option.’ This also does not apply unless an optional Network Interface Card (NIC) has been added to the z620; I assume the card would plug into a PCIe slot.
9. Save the Changes and exit the Setup Utility. The z620 will then power up to the Windows desktop.
10. It is a good idea at this point to power down the computer. Then power it back up again, tapping on the F10 key. When the Setup Utility comes up, check that the changes above (disabling the AMT, disabling Network Boot, etc.) are still in effect.
11. There is one more BIOS step – check and then change if necessary the BIOS boot order. It could be done at this point, but I would recommend doing this after installing the Adapter-NVMe Card.

Note: I did all the above steps, then turned off the computer, unplugged it, and put in the NVMe card here. Why here? So the computer could then power up from the OS already installed and with all the BIOS changes. I want to make sure the computer ‘sees’ the NVMe card with those BIOS settings and a working Windows 10 before I do the next step.  If I install Windows 10 onto the NVMe card at the same time I install the NVMe card, then I have two variables - the NVMe card installation, and the Windows 10 installation.  Should there be a problem it will be more difficult to figure out what is wrong with two major variables than with one.  

[Buffalo_Rider]

Step 6: Install the NVMe SSD card

          It is probably a good idea 💡to Install the Samsung driver for NVMe cards (available on the Samsung support website) before installing the 970 EVO Plus (970 EVO, 970 EVO Pro, etc.).  I didn't do this so I can't say for sure, but it might prevent 5-2 below from happening.  I installed my NVMe card and Windows 10 found it with a Microsoft (generic?) driver.   I did not install Samsung's NVMe driver until after I had installed Windows 10 on my 970 EVO Plus and my z620  was already booting from the 970 EVO Plus.  

1. Turn off the computer, touch your finger to the computer’s metal case to release and ground any electrostatic charge in your body, then remove the power cable from the computer.
2. Install, in your computer, the Adapter card containing the NVMe SSD card; install it into a PCIe slot for that Adapter.
3. Reattach the power cord.
4. Power up the z620.
5. As the computer powers up, the BIOS will detect the Adapter-NVMe Card. A couple scenarios could happen.
   1. The computer, after detecting the Adapter-NVMe Card, powers up all the way to the Windows 10 Desktop. If so, the installation should be good.
   2. The computer, upon detecting the Adapter-NVMe card, gives a 928 Error Message. This is what happened in my case. The Error message said “Fatal PCIe error; PCIe error detected Slot # [where # is the slot number for the card]; Malformed TLP.” At the bottom of the black screen will be an “F1 Boot” in blue. I pushed the “F1” key and my z620 then completed the power up to the Windows 10 Desktop.
6. When the z620 gets to the Windows 10 Desktop, verify that the computer ‘sees’ the NVMe card. Bring up the System Information window. While there, go to “Components->Storage->Disks.”   On the right half of the screen, the computer will show all the ‘Disk Drives’ that the computer is seeing. Mine showed 'Samsung SSD 970 EVO Plus 500GB' as one of those disk drives. This was my NVMe card, so now I verified the computer ‘sees’ it.
7. [Optional] One could also (and probably should also) go into Disk Management to see the additional ‘disk’. The DM will not provide details of the disk drive, such as its name, but it will give you’re a disk number, and you will want to know this when you install Windows 10. During the installation of Windows 10, it will give you options.   It you have other Disk Drives in the z620, you will need to know which one is the NVMe card.
8. Power down the computer

[Buffalo_Rider]

Step 7: Install Windows 10 onto the NVMe SSD card.

This step, like many of those above, is copied from a post of campdublanc and then modified as needed for the z620.

1. [Optional, but recommended:] As a precaution, while the computer is OFF, remove or disconnect all other disk drives that are plugged into the computer. With only one disk to write to, Windows 10 cannot be installed on the wrong drive, and it ensures that all the Operating System parts get installed on just the NVMe card.  If there are two physical ‘disk’ drives in the computer (an SSD Drive is a ‘disk’ drive in this sense), the Windows 10 installer may (will?) write the Operating System on one disk (Disk 1) and the System Reserved parts on another disk (Disk 0) – I believe this is the default on an MBR/Legacy install. I am not so sure about a GUID/UEFI install. But to be safe and ensure the entire OS is on one disk drive, the NVMe SSD should be the only ‘disk’ drive in the computer during the install.
2. Power up the z620 computer and tap the F10 key to get the computer into the Setup Utility; in order to check the boot order in the BIOS.
3. When in the Setup Utility, go to “Storage->Boot Order.”
4. In there, make the “USB Hard Drive” the top most entry in the UEFI section. The hierarchy should start out as “UEFI Boot Sources”, “Hard Drive”, “USB Hard Drive.” Do not change any settings in the lower “Legacy Bios” settings; leave as is.
5. Then do a “Save Changes and Exit.” The computer should finish power up to the Windows Desktop.
6. Shut down the computer.
7. Insert the Win10 USB Drive into the target system's USB 2.0 or USB 3.0 port (depending upon the drive is USB 2.0 or USB 3.0). I used a 32GB USB 3.0 Flash Drive.
8. Insert the REFIND USB Drive into a USB 2.0 port. {Alternatively you could also attach the Drive to an ‘internal header’ and plug the header onto a 9-pin USB connector on the motherboard, or attach it to a ‘USB 2.0 to motherboard cable’ and plug the cable onto the motherboard. See instructions below for how/where to do either of these.}
9. The system should read the REFIND code and bring up its Automatic Boot screen – light gray background, with three colored circles on it. A couple seconds later it will read the Win10 USB Drive and bring up the Windows 10 Setup Window starting the Windows 10 install process. Go through the windows/screens. When offered, select “Custom Installation.”
10. When you are prompted for the Product Key you have two options. If you had Windows 10 on your computer and you are reinstalling it onto the NVMe SSD, then I would click on “I don’t have a Product Key.” Your computer and its Product Key are registered with Microsoft, so the Product Key will be automatically inserted later. You can, however, enter your Product Key if you prefer.
11. If you have multiple ‘disk’ drives attached to the computer: when you get the screen to select the ‘disk’ for the OS, select the one that is your NVMe SSD card. If your only ‘disk’ drive is the NVMe SSD card, this is easy – select it.
12. If the ‘disk’ drive you selected (the NVMe SSD card) has partitions and files on it, delete the partitions so the entire disk is ‘Unallocated Space’.
13. Click Next. It will bring you to the next screen and you should see Windows 10 files being installed.
14. The z620 will automatically reboot when all the Windows 10 files are written to the NVMe SSD card. The REFIND automatic boot screen will now (on this boot) have a 4-pane Windows logo on the left, and the three circles (you saw before) on the right. The computer will then bring you back to the beginning Windows 10 Setup screen.
15. Remove the Win10 USB Drive.
16. Click on the red “X” of the Windows Setup window (upper right corner). Windows will ask if you want to cancel the Win 10 Install. Click ‘YES’ and the system should give you a ‘Getting Ready’ message and then reboot.
17. On this reboot, the REFIND automatic boot screen will have only the 4-pane Windows logo (on a light gray background) signifying, I believe, that on this boot, it booted from the NVMe card. It will then proceed to finish the Windows 10 install by asking you, the User, for input with various setup parameters.
18. Your Windows 10 Installation should now be done.
19. Go into the ‘Disk Management’ Administrative Tool. You should see two disks. One Disk will be the NVMe card with the Operating System Installed on it, which is the C: drive The other Disk will be the REFIND USB Drive – it will say DUET in the Disk title.
20. Click on the Duet disk. Then right click on it and select the drive letter option. As campdublanc stated in one of his messages, you can remove the drive letter. This disk – Duet – will then be hidden, so someone else will not see it from the Windows Explorer, and cannot therefore accidently access it and corrupt it. You can always go back and see it in ‘Disk Management’ if there is a need for that.

[Buffalo_Rider]

Step 8: [Optional, but a good idea] Plug-in the Refind USB Drive Inside the computer.

The reason to do this is so the USB Drive is ‘hidden’ from users, to ensure it won’t be removed.

1. The REFIND code will be read from a USB flash drive through a USB port. One can plug the USB flash drive into a USB port in the back or front of the computer, or alternatively, internally.
2. To connect the USB memory stick internally: Find an internal USB 2.0 port/connector on your motherboard (you will need this later). The motherboard connectors have nine (9) male pins – two rows of pins, one 4-pin row and one 5-pin row. Each connector accommodates two (2) USB 2.0 connectors of 4 pins apiece. (The 5th pin in one row is an additional ground.) For the z620, the three 9-pin USB 2.0 connectors are on the bottom edge of the motherboard (mine are blue in color). The Maintenance and Service Guide (p.26 for the Z620) shows the location. On the z620, there are no Internal USB 3.0 ports/connectors available. Here, as ‘2’ in the following picture, is the pin-out for the cable (or female) side of the 9-pin connector that plugs onto the Z620’s motherboard connector, and here, as ‘1’ in the following picture, is the pin-out of the motherboard connector – you will need this to know how to plug a USB 2.0 internal cable onto the motherboard’s 9-pin connector.

For more information, in general, on motherboard internal USB connectors, here is a good resource and is where I got the above picture: “www.frontx.com/cpx108_2.html “

• Connecting the Cable: the cable has a 4-pin female connector to plug into one of the two rows of the 9-pin USB 2.0 connector on the motherboard. Use the diagram above to figure out how to plug the cable onto the motherboard. Plug the REFIND USB Drive into the other end of the cable. One such cable: Tripp Lite P/N: U024-06N-IDC,
• Connecting a USB 2.0 internal header: they come with either an 8-pin connector or a 10-pin connector (with one ‘hole’ plugged). The 10-pin connector (with a hole plugged) can only plug on one way. The 8-pin connector should be marked on the PCB (+5V, etc.) and from this and the diagram above, you can figure how to attach it to the motherboard. Attach the REFIND USB Drive to one of the USB connectors on the header, then plug the header onto one of the three USB 2.0 motherboard connectors. One such internal header (single layer): ADPU2DU-NO3.

[Buffalo_Rider]

That is it.  I hope the above works for some of you other folks who want to push the limits..  

[DGroves]

one thing that you mention is the boot device, i said SOME win 10 OEM dvd's will not work due to the customized install the OEM uses the Microsoft win 10 DVD will work as it's the "reference standard"

BTW, nice write up, should help anyone who wishes to use nvme on a zx20 workstation