Z620 Z420 Z820 - Boot Block software only flash / easy ME upgrade to 8 / Crisis Recovery useless

[bibikalka]

I wanted to enable ReBAR on my Z620, and chased a few things around, including the BootBlock guide:

https://github.com/SuperThunder/HP_Z420_Z620_Z820_BootBlock_Upgrade

Another item that I became curious about is FPT tool, and its manipulations with the Management Engine:

https://github.com/mostav02/Remove_IntelME_FPT/tree/master

I was also able to track prior HP BIOS updates for Z620. It looks like BIOS v1.x had ME 7 and BB 2011, BIOS v2.x had ME8 and BB 2011, and BIOS v3.x (v3.96) had ME8 and BB 2013. Moreover, 2 packages were offered to flash fresh ME 7 or ME8 using MEBLAST tool, offering one to upgrade to ME8 if desired:

sp59990 (ME7, BIOS v01.14), sp59991 (ME8, BIOS v02.08).

Curiously, in the MEBLAST pdf they have a screenshot of a workstation with BB 2011 and ME8!

So I tried to write my modded BIOS to the chip, but could not pass verification stage of dosflash.exe . So I started chasing the info on the FPT tool.

I did everything in DOS, booting off a USB key. Used a DOS version of md5 sum, and screen capture, to QC the bin files, and saved messages.

First, booting with green jumper moved to BB pins, and FD unlock, I was able to read all BIOS with FPT, but not write either the BIOS area or the boot block, getting 

Error 7: Hardware sequencing failed. Make sure that you have access to target flash area!

So I went ahead and updated to ME8 using MEBLAST in DOS:

*** Override Jumper is installed.
*** ME Unlocked via FDO Jumper.
*** Warning - SPI Flash Regions are Locked.

Opening File = j61_me8a.bin
Reading file......100% 01200000
Flashing Management Engine (ME) Region.
Programming.......100% FF500000
Flashed Management Engine (ME), Reading ME Region, again.

Switch /GlobalValid
Setting Global Valid.
An A/C Power Cycle is required for the ME to load the Variables.
Flashing 'Fixed Offset Nvar'.
Programming.......100% FF000000
ME Firmware update Sucessfull,
Please remove FDO jumper and Power Cycle the system.

ErrorLevel = 0

So indeed, the internal SPI is still read only. Here, I warm rebooted back to BIOS, having v3.96, and ME saying "disabled". I tried FPT again, read only. I ran fpt -greset, it rebooted instantly. After reboot, SPI is still read only. At this point I downgraded BIOS to v2.07 (sp59186) using the standard DOSFlash, and warm rebooted again. Throughout, I kept the jumpers where I moved them. After this warm reboot to v2.07, it booted back, and was saying
MANAGEMENT PLATFORM (ME) IN MANUFACTURING MODE

I go to try flash block update (I cropped the BB region properly), and it goes through!!!

fpt -F B13O396.BIN -A 0xFF0000 -L 0x010000

Platform: Intel(R) Patsburg Chipset - Reserved DID 0x1D41
Reading HSFSTS register... Flash Descriptor: Valid

--- Flash Devices Found ---
W25Q128BV ID:0xEF4018 Size: 16384KB (131072Kb)


- Reading Flash [0x1000000] 64KB of 64KB - 100% complete.
- Erasing Flash Block [0xFF1000] - 100% complete.
- Programming Flash [0xFF1000] 4KB of 4KB - 100% complete.
- Erasing Flash Block [0x1000000] - 100% complete.
- Programming Flash [0x1000000] 16KB of 16KB - 100% complete.
- Verifying Flash [0x1000000] 64KB of 64KB - 100% complete.
RESULT: The data is identical.

FPT Operation Passed

Warm reboot, and the BIOS shows 2013 boot block, same MANUFACTURING MODE. So I first flash the official BIOS 3.96 back, and then on top of it my modified 3.96:

FPT.EXE -F K396TR.BIN -A 0x580000 -L 0xA70000

The flash goes through, I save whatever I can to the USB, and reboot.

At this point it's dark screen, and permanent very loud fans after ~1 min. I tried Crisis Recovery jumper, and it does not work. Tried ~5 different USB sticks. Tried CMOS clearing. Tried disconnecting SATA drives. Etc. USB sticks do blink from time to time, and the logitech mouse blinks too, but for a short period of time. No go.

I pulled out my flash chip clip and Raspberry, but it's tight in there, and the clip keeps getting messed up. I run out of time. Ordered a new clip for ~$2 on eBay, will try once it's back. I do have original full BIOS backups, so having something to flash is not an issue.

Anyway, while being able to flash the 2013 BB relatively easily in software only, I could not test getting out of MANUFACTURING MODE since I got a brick.

I guess modded BIOS is a no go for these machines. And Crisis Recovery does not seem to work in MANUFACTURING MODE. There was another post on that here:

https://h30434.www3.hp.com/t5/Business-PCs-Workstations-and-Point-of-Sale-Systems/Z820-ME-Firmware-q...

Any thoughts/suggestions from anybody other than an external flash?

[SDH]

Impressive work; kind of not surprised at the outcome... said with respect.

Next step... used version 2 motherboard transplant, and we've figured out what to look for on the right-hand side of the motherboard bar code label. There is a point when the 2-alphanumeric "REV" version changed for the v1 vs v2, and also specific HP part numbers to ensure you get a v2. Plus, you need to be very clear with the seller about what you need to get and why. Let them know you will require a refund if they send the v1 type instead of what they showed in their ad pics.

I used to say it took me 1/2 hour to do a motherboard transplant, but that was rushing. It really is about 1 hour from start to first boot.

Last, you generally never know if the Zx20 recycled motherboard you buy was "branded" for Windows or Linux. Finding out is like Christmas morning when you open presents... each time I hope it was branded for Windows, and also that it had been licensed for/upgraded to W10 so it has a digital license out on the Microsoft license servers for easy upgrade to W11.

[bibikalka]

Happy to see you SDH are still here! I've read a bunch of your older posts on these matters - lots of hard learnt findings!

Yeah, I'll probably end up wasting more time than 1 hour with the clip,  but at least it's a challenge! I clipped another HP system successfully before, so have decent hopes.

I forgot to mention that one of the reasons to mod the BIOS was achieving direct NVME boot, but I guess nobody managed to flash that back either? Only that CLOVER approach with the USB key to bootstrap NVME?

I've daisy chained a few mods together into a single package hoping crisis recovery would bail me out if things go south - older microcodes/NVME/ReBAR, so tough to say if it's 1 change that bricked it, or some checksum issue. If I recover it, probably won't try again.

Apparently, Management Engine is a curse to many people given its opaque nature, so manufacturing mode can be leveraged for some of those objectives.

[bibikalka]

---

@SDH wrote:

Impressive work; kind of not surprised at the outcome... said with respect.

Next step... used version 2 motherboard transplant, and we've figured out what to look for on the right-hand side of the motherboard bar code label. There is a point when the 2-alphanumeric "REV" version changed for the v1 vs v2, and also specific HP part numbers to ensure you get a v2. Plus, you need to be very clear with the seller about what you need to get and why. Let them know you will require a refund if they send the v1 type instead of what they showed in their ad pics.

I used to say it took me 1/2 hour to do a motherboard transplant, but that was rushing. It really is about 1 hour from start to first boot.

---

 Alright, with the 3rd clip in my hands I was able to recover it 🙂

Without removing the motherboard the space is quite tight. It's very tricky to attach the clip, need to use 2 hands from 2 sides so it's vertical without tilt. The clip plastic teeth wear out if placed incorrectly, so I destroyed 2 clips, but really prepared with the 3rd one - and it worked. Many hours - this is more like a hobby, for sure! I learnt a tons of electronics skills. The clips were cheap, but had to wait for them to arrive.

Obviously, for 99% of people replacing with a working motherboard is a much faster solution.

I filled with the glue the little area in the middle of the clip where the pins are visible to the side with Superglue. Otherwise the clip pins had some up or down motion coming out of alignment, which was a bad issue.

There is a pad next to the BIOS chip with useful contact holes - I could test chip connection with a multi-meter first, and also see if Raspberry Pi is providing enough voltage, without big losses. I had about 3.2V on this test pad.

I reflashed an old copy of my full BIOS I made before starting any work, with only boot block updated to 2013. The flash was uneventful. When it rebooted, Manufacturing mode was gone (it was an old BIOS backup!).

So basically to summarize, one can write a fully custom BIOS by triggering manufacturing mode first as I described. The custom BIOS region should be embedded into a working BIOS version with fully initialized ME  (your early copy, no manufacturing mode!!!), such that the jumper recovery mode can kick in if something is wrong.

There is a report of a BIOS modded with the NVME boot module. That BIOS was flashed externally - but manufacturing mode will allow that to be flashed internally as well.

I may play in the future with getting the computer into and out of the manufacturing mode, and will try the jumper recovery too, maybe some custom BIOS later on 🙂

Some more notes are here:

https://www.techpowerup.com/forums/threads/hp-workstations-owners-club.254315/page-13#post-5251044

 

 

[bibikalka]

---

@SDH wrote:

.

Next step... used version 2 motherboard transplant, and we've figured out what to look for on the right-hand side of the motherboard bar code label. There is a point when the 2-alphanumeric "REV" version changed for the v1 vs v2, and also specific HP part numbers to ensure you get a v2. Plus, you need to be very clear with the seller about what you need to get and why. Let them know you will require a refund if they send the v1 type instead of what they showed in their ad pics.

I used to say it took me 1/2 hour to do a motherboard transplant, but that was rushing. It really is about 1 hour from start to first boot.

---

Btw, what is the significance of this v2 motherboard? Just not to have to mess with the Boot Block 2013? Or do you think there are some electrical differences between v1 and v2?

I have a fairly ancient v1 motherboard (J61 V1.02 what the 2011 Boot Block said).  Is that gonna be any issue running a v2 Xeon using the 2013 Boot Block that I flashed to this motherboard?