HP zX20 BIOS v 3.92 > Successful roll back to v 3.91

[BambiBoomZ]

NOTE: This post indicates the method to roll back BIOS 3.92 in HP zX20 systems to an earlier version. The 3.92 BIOS version is strongly recommended by HP to protect the system from Intel processor Meltdown / Spectre vulnerabilities. Reverting the BIOS version to restore performance therefore has an inherent risk. Anyone rolling back should understand the risk and assume the best security measures  possbile including a VPN and continuous high grade malware and virus monitoring.  < Good idea anyway these days.

HPeers,

For the zX20 system, BIOS version 3.92 proved extremely problematic to my principal 3D CAD modeling system, a z620. However, after three months, the problems have been successfully solved.

The ostensible purpose of  the strongly recommended BIOS 3.92 was to protect systems from the Intel CPU Meltdown / Spectre vulnerability to a kind of back channel memory attack. 

I had three HP z-series at the time: z620_2,  z420_3, and  z420_1, all running v 3.91.

In the first instance, the upgrade on z620_2from v3.91 to 3.92- done in the identical way used at least ten times previously, corrupted the BIOS and the system became unusable, even to diagonise the problem.  As the crisis recovery jumper was undocumented in the service manual and not shown on the system board diagram, I assumed that the motherboard was ruined.

A replacement motherboard was purchased and installed, unfortunately again using v3.92.  However, the performance was so poor that I decided the motherboard was defective and returned it.  This happened again with a second motherboard and that mean that v3.92 was reducung the Passmark CPU score from 17128 to 16290, the 2D on the Quadro P2000 went from 876 to 549, and 3D from 8998 to 7640.  I later learned that the performance drop was more pronounced on overclocked CPU's.

Eventually, thanks to a post by forum friend  SDH regarding the crisis recovery jumper, the original motherboard was recovered, but to v 3.92 and the poor perofmrnace.

Over almost a full month, several strong appeals for a simple reply from HP as to the possibility of rolling back v3.92 were ignored. At that time, all the internals of z620_2 were transferred to z420_3 (E5-1620_2) which was running on 3.91, and the Passmark results were not far off z620_2 at it's best which had a top rating of 6322. This is z420_3 replicating z620_2:

Becaase z420_3 replicated z620_2 almost exactly except in the BIOS version, these results comfirmed that 3.92 had noticeably degraded system performance. For a 3D CAD system, the CPU Single Threaded is the most important score in this test and 2373 is by a bit, the best result ever for any of the five z-series systems. The average single threaded mark for an E5-1680 v2 is 2110. The very good Disk Mark is courtesy of a first generation HP Z Turbo Drive 256GB M.2 AHCI. The Z Turbo Drive must be highly optimized: for comparison, the Passmark average results for a Samsung 960 EVO 500GB NVMe is 14548. These results, by the way, made z420_3 the highest rated z420 in Passmark baselines.

However, z620_2 remained in limbo. Eight attempts to revert 3.92 to 3.91 in about four variations of technique failed. This gave the appearance that v3.92 must contain undocumented microcode that prevented rolling back the version.

After another query, "KelleyC" in business support relayed information that there was indeed microcode preventing 3.92 version roll back. The good news was that there was also a secret, magic button that could allow the version to be reverted to 3.91; a green jumper next to the main motherboard power connection, just above the SATA ports:

 

Remove the green jumper, load the new BIOS and replace the jumper. Within about 15 minutes, BIOS 3.92 was reverted to 3.91.

Three months frustration, many hours, and expense, might have been avoided by simply knowing the function of two tiny jumpers. 

The test results for the restored z620_2 are quite good:

HP z620_2 (2017) (Rev 4) > Xeon E5-1680 v2 (8-core@ 4.3GHz) / z420 Liquid Cooling / 64GB DDR3-1866 ECC Reg / Quadro P2000 5GB / HP Z Turbo Drive M.2 256GB  + Samsung 860 EVO 500GB + HGST 7K6000 4TB / Focusrite Scarlett 2i4 sound interface / 825W PSU /> Windows 7 Prof.’l 64-bit  > 2X Dell Ultrasharp U2715H  (2560 X 1440) 

I think the comparative results for z420_3 and z620_2 which ar nearly as possible the same system except for the motherboard, demonstrate something I've had in mind awhile. The z420 and z629 motherboards appear to identical =- and use the same BIOS, except that the z620 has sockets to mount a 2nd processor riser- and the phantom of that socket is visible on the z420 motherboard.  It seems that, all things being equal, without the extra QPI links to accommodate a second processor, a z420 will slightly outperform a z620. 

Thanks KelleyC in Support for finalizing the end of the v 3.92 disaster and to forum friends Brian1965 and SDH for their advice and  generous patience.

BambiBoomZ

[SDH]

Bambi,

It is great to hear that you have made the breakthrough.  A question for you is down at the bottom.

For those who have not seen my Crisis Recovery Jumper post for the Z420/Z620/Z820 workstations it is HERE.

https://h30434.www3.hp.com/t5/Business-PCs-Workstations-and-Point-of-Sale-Systems/Crisis-Recovery-Ju...

I believe the reason HP did not document it hardly at all is that one can get into trouble using that method, but we have used it as a last resort and it has saved some of us with corrupted BIOS installs.  It seems to have been engineered in for HP service reps to use, and that particular first black jumper (a picture is shown in my post above, and here below) is different from the lime green one Bambi has discovered.  That first jumper  seems to allow recovery from a corrupted BIOS install, but as Bambi found it only works if the BIOS does not have an encoded lock built in.  Almost all older verions of HP BIOS did not have that software lock.... some newer versions do, but HP usually has documented that new circumstance in the documentation available from the same page that one can download the SoftPaq BIOS installer from.  However, I don't see that information documented in the 3.92 BIOS Release Notes for the combined Z420/Z620 motherboards (they both use the exact same BIOS installer).

Now along comes the second undocumented BIOS related jumper that Bambi found..... my black one referenced above is in the interspace just above the bottom PCI slot.  This second one is lime green and is near the motherboard's bottom front corner adjacent to the main power supply's "P1" attachment to the motherboard.  It is above the front case cooling fan's 4-pin motherboard header.  Of interest two Z620s I just checked have that lime green jumper in place, but the Z420 here has the pins but no lime green jumper crossing them.  I checked on eBay for Z420 motherboards and a bunch of them are there with pictures and all have the lime green jumper in place.  Someone in the past must have removed the jumper from my Z420 and forgot to replace it.  This proves that lime green jumper is not critical to proper motherboard function.  I'm guessing that having it in place allows the encoded-in-BIOS lock to function, and removing the jumper disables that microcode software lock.  Or, maybe one needs to remove the lime green jumper before the black jumper tricks will work.  My Z420 was one that got its BIOS corrupted, and maybe the absence of the lime green jumper is why I was successful in finally getting that corruption fixed with a fresh BIOS load from a USB stick.  I don't remember the version, but it might have been 3.92 that got corrupted and I just got lucky having the green jumper missing from the past.

EDIT:  Added info... for the ZX20 family there is a "Password Jumper" you can find in the technical and service manual, and there is a section in there that describes "Resetting the password jumper".  In that section the jumper color is described as blue, but the motherboard diagrams show that password jumper to be exactly where Bambi's lime green jumper is.  There is no nearby alternative blue jumper, and the Password Jumper header is described as "E49" for all the ZX20 family.  E49 is printed in tiny letters on my Z420 and Z620 motherboards right at one edge of the lime green jumper's two pin header....   So, it would appear that jumper has more than one function, and is lime green rather than blue.

Here's the question for Bambi..... did you remove the lime green jumper and simply load back in the 3.91 BIOS the normal HP-documented way, or did you remove the lime green jumper and load the 3.91 BIOS using the Crisis Recovery Black Jumper way described in my post above?

It would be great if you were able to just downgrade from 3.92 to 3.91 BIOS the old fashioned way with the lime green jumper temporarily absent.  For me the safest way to update BIOS from within BIOS.  I truly believe that if everyone did it that way almost all of the BIOS corruption events, while already rare, would disappear.  My Z420 BIOS upgrade corruption happened when I took a shortcut and chose to upgrade BIOS from within a W10Pro64 install.  Never again....

And, I'll assume you put the lime green jumper back on.

[BambiBoomZ]

SDH,

Thanks again for contributing the essential crisis recovery jumper technique to the revival of z620_2.

I think HP is somewhat careful in promoting a deeper level of experimentation by users, and anyway workstations are meant to be ultra-reliable and the user isn't supposed to need to get under the hood and fix it. Actually, z620_2 is the first workstation I've ever had to seriously repair- all other work involved varoious levels of upgrades.

However, with obsolete and out of warranty systems- my last HP warranty (z420_2 (E5-1660 v2) ended last February, over time may be increasingly subject to substantial upgrades /modification. As time progresses, there is an increasing need and potential benefit  that all the details of control and service functions become known. I suppose that's why we're here in this forum. I haven't seen anyone with a new Z6 asking questions.

By the way, here is the chaos when transferring the internals from z420_3 back to z620_2:

There was no work surface large enough to do this other than the floor.

The Magic Green Jumper: As to the method of use, the green jumper does appear to be a key to a lock on a certain level of microcode reading. What I didn't mention in the previous post is that KelleyC in Support referred to a second "red" jumper that must have a similar function. "Try the green jumper and of that doesn't work, try the red one." The red jumper turned out to be in fact a black one just above the SATA ports. It seems logical that it would have a function in this realm as it is quite near the CMOS reset button- just to the left and slightly above:

That's the Z Turbo Drive in Slot 4.

Sorry for the poor image quality.

I couldn't comment on the "Mysterious Black Jumper" meaningfully.  As the green jumper was successful  I never tried the black one.

The use of the green jumper is simple: remove the green jumper, load the BIOS using the conventional method of the .EXE through Windows , and then replace the jumper which must be a protection against certain microcode writing. As was the use of the Crisis Recovery Jumper, the Magic Green Jumper method was very fast and almost depressingly easy after so many hours of uncertainty, expense, and effort over nearly three months. Of course, we all wish that the simple answer would appear in the first half hour!

I hope that MtothaJ, who has done microcode modifications to allow NVMe support will comment. 

BambiBoomZ

[SDH]

Bambi..... good to know. 

Here is the warning from the latest Z400/Z600 3.61BIOS upgrade that HP should have had in the ReadMe document for the latest 3.92 BIOS for the Z620 (but we know how to get around that now):

"HOW TO USE: 
WARNING: After installing this BIOS version onto the system, prior BIOS versions cannot be installed onto the system."

I have an update on the Crisis Recovery Jumper process.  It turns out that HP had an early BIOS updater for the ZX40 family of workstations (such as the Z640) and that it bricked a number of systems with same symptoms we have seen in the ZX20 family rarely.  HP put out a customer advisory that describes proper use of the Crisis Recovery Jumper, also known as the "Boot Block Recovery Jumper",  HERE.

https://support.hp.com/us-en/document/c05163478

I've described how to locate the .bin file of the BIOS you want to install/downgrade to in the forum here.... it is inside the DOS folder that gets created inside the SoftPaq folder inside the SWSetup folder on the root level of the C drive when you run the SoftPaq and cancel the process after you get barely into the process.  That SWSetup directory stays on  C drive available to harvest the .bin file from.

Here's the path, but please note that this shows you the path to 3.92.  Same idea for if you are harvesting the 3.91 BIOS .bin file (which is where I advise to stop until HP fixes the mess that 3.92 may create).

[DigitalYemeni]

This BIOS update screwed my day! After I got a new motherboard I was stupid enough to go ahead and update it with this BIOS version since HP strongly recommended it! 😛

Once the PC rebooted, it went into a BIOS loop and couldn't find a way to get in! only HP splash screen loogo shows up then it restarts over and over! Pressing the F10 or F9 or whatever display that entering the setup.... then poof! reboot again and again!!

I had to clear the CMOS then i managed to get it FINALLY! I figured out that when the setting is changed to AHCI which was initial was the setting during the upgrade, caused this BIOS boot loop!

I just downloaded v3.91A and installed it over (thru Windows 7 exe) then it came back to normal with the old crappy 3.92 wiped off!!

HP, for godness sake, please REMOVE this BIOS update from the download section and save people the heartattack!!!!

[SDH]

There is a working HP link to the Z620/Z420 3.91 BIOS  HERE, and note that the Z420 and the Z620 workstations use the exact same BIOS installer and version.  The Z420 motherboards are a slightly stripped down version of the Z620 motherboard, but are not perfectly interchangable on a hardware level.

https://support.hp.com/us-en/drivers/selfservice/swdetails/hp-z420-workstation/5225033/swItemId/wk-1...

For those who need to harvest the .bin file for use in the BIOS roll-back from 3.92 to 3.91 you'll download the SP78208, run that on any computer, cancel the process early on at the first splash screen that opens, navigate to the root level of your C drive, find the SWSetup folder with the SP78208 folder inside, go into that and find the DOS Flash folder at the top, and copy out the J61_0391.BIN file onto the top level of a small thumb drive.

This is the same harvest process I use to get that .bin file to use during upgrading BIOS from within BIOS, but the overall use of that thumb drive is different for that process than the BIOS roll-back process.

[theOne77]

will this rollback of BIOS will possible to HP Z400?

[SDH]

I have updated our Z600 and Z400 workstations (all v2 motherboards) to the latest HP BIOS and have not had any problems. 

The one bad experience I had was from updating to the latest BIOS in a Z620 from within Windows 10Pro64.  That bricked the motherboard and led to the search for the methods to reflash a corrupted BIOS.  Then Bambi got key info from HP on the lime green jumper trick.  So, there are the black jumper and the lime green jumper.

Both exist on the Z400 and the Z600 (and on the Z800 I'm sure too).  Same methods should work for those workstations too.

[theOne77]

SDH so if i will try to rollback my BIOS in HP Z400 there will be NO ISSUE if i follow the same procedures?

[SDH]

Dude.... there are no promises here.  At my house I'd say yes.....

This procedure is technically sophisticated, but well in the range of what many of us can do.  The Crisis Recovery procedure is a level of complexity above this, and a number of us have pulled that off too.

If I had a bricked motherboard I'd run that crisis recovery process today. 

I have a number of the ZX00 and ZX20 boxes that have been updated to the latest BIOS and I'm not rolling those back, personally.  Maybe later.  They're working just fine.  But I'm not updating any more, yet.  Our friend Bambi is in a whole other league than me in terms of need for speed.... he can see some slowdown from the latest BIOS.  My use of these workstations actually mandates that I slow down and look, so my need for speed is less.