Crisis Recovery Jumper Z620 Z420 Z820 Revealed

[SDH]

EDIT:  One thing not mentioned below is that I use my usual methods to clear the CMOS before I proceed on to the dump/reload of the BIOS using the Crisis Recovery Jumper method.  I recommend doing this as your first step because that will reload the Factory Default BIOS settings.  What if there are two things going on?  CMOS might be corrupted or set incorrectly, and this first step would fix that.  Then the Crisis Recovery Jumper method deals with a much deeper issue....that BIOS itself, not just its settings, is corrupted..... and needs to be dumped and reloaded fresh from a CD or USB drive.  Some BIOS corruption issues may not be fixable, too.

There have been questions on this forum about where is the Crisis Recovery Header/Jumper for the Z620 Z420 and Z820 workstations.  First.... what is it?  On these HP workstations It is a motherboard header comprised of 3 pins which has a 2-pin jumper associated.  When the jumper is "loaded" or "shorting" two specific pins it is in its normal running state.  If the jumper is  shifted from its default position to short the other two pins it will initiate a semiautomated dump of a BIOS that is corrupted and allow the load of a known good non-corrupted BIOS from a CD or thumb drive.  This may, in some cases, allow a workstation to be revived from an otherwise fatal condition.  This process is fixing an issue deeper than simply having incorrect or corrupted CMOS settings.

It is clearly a very advanced procedure, as you'll see.  Don't forget to move the jumper back to its default run position when done.  My advice is to ignore this post unless you really have a dead end condition for your workstation.

This header/jumper is minimally documented in the case of the Z820, and not documented in the Service Manual for the Z420 and the Z620.  I have salvaged a Z620 with these techniques, as have others.  The post below is the best I can offer at this time, and it is hard to justify doing this procedure over and over to get the details more precise.  Here it is:

EDIT:  HP has provided a more detailed procedure description for its newer Z440/Z640/Z840 workstations as a customer advisory, linked down this page.  Also, it appears that HP has configured the Z820 and Z840 workstations opposite to the others in jumper placement.  Shorting pins 2-3 is the default for normal running of the Z420/Z620/Z440/Z640 workstations, but for the Z820 and Z840 the shorting of pins 1-2 is the default for normal running.  Wonderful...... 

Excellent post, from Dan_WGBU, our favorite HP engineer:  LINK

https://h30434.www3.hp.com/t5/Business-PCs-Workstations-and-Point-of-Sale-Systems/Jumpers-on-Z620-FM...

This post gives added details from Ashleight-Paul on the process, however note that he is working on a Z820 and thus his pin positioning of the jumper is opposite that done with a Z420/Z620.  LINK

https://h30434.www3.hp.com/t5/Desktop-Boot-and-Lockup/Z820-Bricked-Bios-after-Firmware-update/m-p/57...

This post details the guy doing the process incorrectly and failing:  LINK

https://h30434.www3.hp.com/t5/Business-PCs-Workstations-and-Point-of-Sale-Systems/z420-BIOS-v-3-91-u...

Do a Google search for “669531–005” to find the latest ZX20 technical and service manual, and download that.  Version 5, the last one published, is the one you want, and this link works:  LINK 

http://h10032.www1.hp.com/ctg/Manual/c04205252

Z420 & Z620:  The header designator is E15 = ROM bootblock recovery header = “E15” printed on motherboard at top left of jumper.  The position of this jumper for the Z420/Z620 is not shown with a number or a marker on the service manual's MB diagram but it truly exists (see picture below).  In contrast, the Z820's MB diagram's #39 in its section of the service manual shows where the jumper is located.  It is in the exact same position for the Z420 and Z620 (which have almost identical motherboards and use the same BIOS).  I have no idea why the service manual shows the jumper position for the Z820 but not for the Z420/Z620.

Z220 & Z820:  Designator is E15 = BIOS Crisis recovery header/jumper = “JP15” or “E15” printed on motherboard.  It is #29 on Z220 SFF motherboard’s diagram; #34 on the Z220 CMT MB’s diagram; and #39 on the Z820 MB’s diagram.

Included below are my clarifications for the Ashleigh-Paul post linked above:  Please note that my experience with this process was somewhat different than his, but the basics are here, and this did save the Z620 I was asked to salvage.  It would not boot into BIOS, and after a delay (with only a black screen showing) all the fans then ramped up to full speed and stayed that way.  Now it is back to normal function.

1. In the ZX20 workstations there is a motherboard header, E15, incompletely documented in the Technical and service manual, which can be used to force load of a BIOS .bin file from a USB drive into the motherboard. The position of this 3-pin jumper is partially documented in the technical manual, but the position for the Z420/Z620 virtually identical motherboards is the same as its position on the Z820 motherboard.  The Z220 SFF and CMT motherboards have two different positions, which are documented in the manual.  There is a big difference between a restart and a cold boot (from power fully off).  You must follow the instructions below exactly:
2. Power off the workstation and unplug the power cord.
3. CORRECTED:  Change the jumper position from bridging pins 2-3 (the default condition) to bridging pins 1-2 (the Crisis Recovery position). In the Z420/Z620.  The E15 motherboard header is between the bottom PCI white connector and the next up PCIe black connector.  Pin 1 is the far left (towards the workstation's backplane).  Remember that the header position and pins to jump are different for the Z820.
4. Have a non-NTFS formatted small USB2 drive with the target BIOS .bin file on it, at the top level of the drive, and plug that in to the TOP FRONT USB2 port of the workstation. You could also use a rear USB2 port.  Do not use a USB3 port.  For the current 4/18 BIOS for the Z420/Z620 version 1 and 2 workstations the BIOS is 3.92, and the .bin file you would have on the thumb drive should be named J61_0392.bin.  This can be harvested from the BIOS SoftPaq by auto-unpacking it and going into the SWSetup folder that creates on the root level of C drive, and then going into the source SP84163 folder that is created in SWSetup, and go into the DOS folder that is in there, and copy out that .bin file.  You can turn off hiding of extensions of known file type in the Folder view tab control panel to make this easier to find (I keep that setting unchecked always).
5. Plug in power cord, power up, and wait.... you should see it blink the USB stick on, and then 7 red lights/beeps from the workstation. NOW wait.... the workstation will auto-reboot, DON'T TOUCH ANYTHING!
6. It will restart and you will get blinking USB Stick, and red lights flashing (but this time 😎 and then a few seconds later the workstation's lights will go blue and the BIOS will come on. DONT TOUCH YET!
7. The workstation will restart again, and the BIOS screen will start, NOW PULL THE USB STICK OUT, and you will see BIOS firmware flash itself.
8. The workstation will now auto-restart, and flash 9 times and then fully auto-shut itself off.
9. Unplug the power cord, and don’t forget to set the recovery jumper back to its default position (bridging pins 2-3).  That is, shift it back to the far right on the pins, where you found it before starting the process.
10. Plug back in the power cord, power up, and restore any settings you had in the BIOS, and BIOS should be back to working properly.

Again, the number of beeps likely varies with the particular problem(s) that this advanced firmware mechanism encounters, as it did in my case.  This is a last ditch effort procedure so I had no hesitation to keep experimenting until I succeeded.

Here is a picture and an engineering display of the crisis recovery jumper present on all Z420/Z620 motherboards:

Jumper positions:

                                                                     1   2    3

Rear of computer                                  .     .---.                   (default)                                     Front of computer

                                                                     .---.     .                   (during recovery)

6/27/18 EDIT:

It turns out that HP had a bad BIOS released for the ZX40 series of workstations (such as the Z640) and that bricked a number of the workstations it was applied to.  HP released a customer advisory that includes the official steps to take to use the BIOS Recovery Jumper, and this method appears to be what you'd use on earlier HP workstations.  The advistory is available  HERE.

https://support.hp.com/us-en/document/c05163478

Advisory: HP Z440, Z640, Z840 Workstations - System Won't Post to BIOS And Or All System Fans Get Loud | HP® Customer Support

RESOLUTION

-The fix for this issue is BIOS revision 2.20. On any systems not in the failing state BIOS revision 2.20 should be used to update your system BIOS.
-If your system gets is in the failing state you can call support to walk you through the instructions below to attempt to recover the BIOS or personally perform the steps below:
Steps to attempt to recover BIOS:
1. Power off system and unplug power cable.
2. Remove side panel and locate "Boot Block Recovery Jumper" on side panel label.
3. The jumper will be on a 3 pin header where 2 of the 3 pins will be loaded by the jumper.
a. On the Z840 load the jumper on the 2 pins towards the front of the chassis.
b. On the Z440 and Z640 load the jumper on the 2 pins towards the rear of the chassis.
4. On a USB key (preferred format FAT32), with a built in LED to show activity, place the .bin file of BIOS version 2.20 or later install on the root directory.
5. Install the USB key onto the failing system, insert power and press power button.
6. When the system is powered on it should be followed by 8 beep audio code. Disregard this as this is normal behavior.
7. The USB key activity LED will begin to blink slowly then rapidly in most situations. This process will occur twice.
8. If BIOS Recovery is successful, the display will show a BIOS Recovery screen with the update in process. "Don't shut off power at this state."
9. When install is complete, the system will power off.
10. At this point unplug the power cable and USB key.
11. Return the Boot Block Recovery jumper to its original location. Replace side panel.
12. Insert power cable and press the power button. The system will come up with a black screen and shut off once, then power on by itself again and if successful the BIOS post will appear.
*If recovery steps unsuccessful replace system board.

Note that last line..... our team has found that sometimes it takes multiple attempts for the BIOS recovery process to work, so don't give up on the first try.  You may also want to remove the "Password Jumper" during the attempt(s) because that may allow a corrupted locked BIOS to be recovered.  Earlier BIOS builds were unlocked but more recently they have been released as locked after update.  A lock code can be built in by HP that is intended to prevent downgrading of BIOS to an earlier version after an upgrade has been done.  The Password Jumper being absent allows unlocking that lock.  Don't forget to put back the Password Jumper when you are done.  More details on this issue are deeper into this post.

[SDH]

The main post has been edited and corrected.

[werkkrew]

I am in a similar situation with a Z820 that turns on, full speed fans, solid blue light.

I followed the process outlined in this post and had some luck - but still no joy.

I put the 3.92 bin file for the bios on a thumb drive and boot with the crisis recovery jumper set.

I get the reboots with the sequence of beeps/red light and I am even able to get to the point where I can see the boot splash screen appear on the monitor.

It gets to a point where it says "Flashing System Rom, do NOT power off" in a blue box but it stays at 0% forever.

I have left it for a few hours and still 0%.

Luckily I can power it off and repeat this process but I can't get anywhere beyond this point.

Any ideas would be greatly appreciated!

[BambiBoomZ]

werkkrew,

If you read elsewhere on this forum, there are a number of my posts concerning the extensive problem caused in my current  z620 by BIOS v3.92.

Intitially, when I updated 3.91 to 3.92 about a week after it''s release, using the same method from perhaps ten times on  five HP z-series systems, the BIOS was corrupted and the I thought the board was ruined and bought a replacment.

After finding out from SDH on this forum that the z620 has a crisis recovery jumper - the motherboard was recovered.  In the z620 User Manual the recovery jumper is not mentioned nor shown on the  system board diagram. The jumper appears to be listed only for the z820, but with no mention of how to deploy it.

After the BIOS failure that I thought had ruined the motherboard, I went to an extreme in preparations: formatted the USB in FAT32 (as recommended), set a 1GB partition on the 32GB drive- that should really be 512MB or less according to some, removed all the system drives, changed the z620 CPU from an overclocked 8-core (E5-1680 v2 @4.3GHz) to a non-hyperthreading Xeon (E5-1607 v2 4C@3.0GHz), ran 1X 4GB of RAM, and the GPU was a fanless 256MB Geforce GS7100 running a generic VGA driver to a low resolution monitor. CMOS had been cleared by having the battery out an entire day. During the process, I had a print-out of the instructions in front of me that I'd separated into numbered tasks. Perhaps the preparations were obssessive but the process worked on the first try. 

On some level, I think the key factor is the timing of removing the USB drive. The reason may have to do with the idea that the bin file should be read off the USB, but that running the USB drive after writing to the BIOS causes trouble. There may be, in effect, a time gap in which the system has no working BIOS while there is a recognized drive still operating. < That's a guess,..

But certainly, give it as another go, but I strongly, strongly, strongly recommend using v3.91 to avoid the noticeable performance reduction of 3.92 and that, for whatever reasons, caused what for me has been the most disastrous and expensive computer problem in 20 years. The v 3.92 may or may not correct Spectre / Meltdown, but I never had such a destructive attack on a system from a hacker as from v3.92. I should mention also that this morning was the 7th unsuccessful attempt using seven variants of the methods to revert the 3.92 to 3.91.

BambiBoomZ

[werkkrew]

Thanks for the detailed reply.

I am wondering if at any point during your efforts you experienced the same thing I am with the "Flashing System Rom" display freezing at 0%?

I have tried removing the USB stick at various points in the process and noting ever changes.

Starting to face that reality that I might just need a new motherboard.

[SDH]

I'd keep working on it.  Bambi now has more experience with this than I do but I did go through multiple attempts before I got the right combination.  And, by that time I was not paying a lot of attention to details so I don't know the exact key to the process.

If you happen to have a version 1 Z820 you might want to switch over to a version 2 motherboard if it comes to needing a new/used motherboard.

[BambiBoomZ]

werkkrew,

I n all my various flashing failures, I never had a 0% error, but instead always had an erroneous report of 100% success- that is, until the last 3 seconds.

Consider the details of all the components involved: ensure the USB drive has the proper format for the function. I saw several mentions to use FAT32 as NTSF will not work , also: the partition ishould be a minimal size. I've seen comments that it should be no mo re than 512MB, I think it saves a full size copy of the original, reads the new version which it may store in the ROM, and then erases the original.  It's possible that the BIOS ROM , needs to accommodate in effect two copies, but if the write drive is too large, files could be spread over an area on the disk and may not be able to read it all at once.  Also, disconnect everything possible in terms of USB, Internet, and drives, and disable all the startup items that makes sense,  so in combination, the I/O to the CPU is minimized. Don't touch the K/B while it's reading and writing.

Some of this reasoning is speculation, but based on collecting comments and recommendations. I like to use the stictest procedure around BIOS.  "Once burned, twice careful" is the saying or in my case; "Seven times burned, about to sell it on Ebay."

Die Zukunft is jetzt,

BambiBoomZ

[werkkrew]

@SDH - It is a Rev 3 motherboard (1.3) so it's about $100 more to replace than the other ones.

@Bambi -

Yeah I will try to partition the drive smaller but the fact that it boots from it at all makes me feel like the USB device is ok.

Right now I am in the process of trying it from a CD.

[ShadowX360]

What BambiBoomZ said about the partition size is key!!!!! I did this over a dozen times scratching my head why it wasn't working even though it was reading off the drive until I read this post and grabbed a 1GB drive. Worked on the first try.

[SDH · Accepted solution]

ShadowX..... please clarify.  Why "1GB drive"?

That seems way too big.  If you have the .bin file from the latest Z420 BIOS installer you can see that it is only 16.5 MB large.... J61_0392.BIN... 16,384 KB.  Let's say the old BIN file is same size.... so for the process to copy out the old (corrupted) BIN file onto the USB drive plus use the new BIN file stored on the USB drive all  you'd need is a 32MB thumb drive... plus a little more for wiggle room.

Why did you leap to 1GB?  Was it that or was your success just from good juju?

However, if this is the key to success for others too then we'll accept it and heap praise on you.