HP2000 MSA G3 has got LEFTOVR disks

[www_tank]

Hi all!

Unexpectedly my HP2000 MSA G3 has got a disk with LEFTOVR state.

The vDisk is in CRIT state and it is doing “RECONSTRUCT”. I set a spare disk in this vdisk. Perhaps all data from LEFTOVR disk are removing to spare disk.

It was once when all volumes were empty. That time I deleted and recreated all vdisks (and also volumes). Now I have data on it! Nothing happened which could initiate any problem. But I have received alarm!

HP tech support gives only mistakes for “Troubleshooting Steps for Leftover Drives" inquires.

[DCESStorage]

Re: HARD DISK - LEFTOVR

Generally I don't recommend taking this approach without going thru HPGT.

Also - you should most likely consult HP - to exactly understand the events and extent-of changes that occured which may have caused the current situation. Also, you have good backups, right !?

Now - if you exhausted all the above options - just a quick process we worked recently upon a power-hiccup, and LEFTOVER disk (amber-light) event. We know this LEFTOVER disk did not move from some other slot, nor it came from some other MSA/system. We are sure - this LEFTOVER event occured due to a power-hiccup/event. Also, we had the RAID/Vdisk has not-failed (ofcourse - in critical condition though!), and hosts are still accessing the volumes on that vdisk. With this backgroud, here is the process/detailed-steps. You've gotta have Word or OpenOffice for this to read 🙂

How to rebuild a degraded vdisk in a HP MSA P2000 / 2012fc; A disk has changed State to “LEFTOVER”; original spare disk showing state: “AVAIL” ; RAID5 become CRITICAL!

We had a power-event and MSA had a power-hiccup. It is possible some disks came-back online AFTER controller came-up. Initially – we could not get to MSA controllers either thru SMU/Web-interface or thru putty, or much worse – not even via the USB cable hooked-up from controller-A to Backup server. After moving the USB cable to controller-B (bottom) now we regained access thru serial port (COM3) via putty. Restarted management interfaces of both controllers, and now we can communicate with the controllers via putty, and via SMU/Web Management Interfaces.

Checking thru SMU, Narrows MSA P2000 indicated VDisk Failure, reporting following messages:
Degraded.
Virtual-disk is not fault tolerant.

 Event 55: A disk drive reported a SMART event
 Event 314: There is a problem with a FRU.

The Physical Hard Disk Drive likely not failed, but the vdisk array was in a degraded state. One Physical disk (Enclosure#2, disk#4) was lit AMBER – indicating some sort of failure/malfunction.

The first thing that needs to be done, is the meta-data on the leftover disk needs to be cleared before it can re-join the RAID as either a regular disk or a dedicated spare.

Clearing Disk Meta Data

Each disk has metadata that identifies whether the disk is a member of a vdisk, and identifies other members of that vdisk. If a disk’s metadata says the disk is a member of a vdisk but other members’ metadata say the disk isn’t a member, the disk becomes a LEFTOVER. The system overview and enclosure overview pages show the disk’s How-Used value as LEFTOVR. A leftover disk’s Fault/UID LED is illuminated amber.

Before you can use that disk in a new vdisk or as a spare, you must clear that disk’s metadata.

To Clear meta data from LEFTOVER disks

1. In the Configuration View panel, right-click the system and then select Tools > Clear Disk Metadata.
2. In the main panel, select disks to clear metadata from.
3. Click Clear Metadata. When processing is complete a success dialog appears.
4. 4.  Click Ok

Re-Join Physical Disk to VDisk RAID

1. In the HP MSA Storage Management Utility, Right click on the affected vdisk and select Configuration > Manage Dedicated Spares
2. Your disk should appear in the list of available drives with a state of AVAIL. Tick the drive, then click the Modify Spares button.
3. The disk will be re-joined to the array. Initially it may be listed as a spare, but the MSA will automatically re-join the disk as an active member of the RAID if this is how it was originally configured.
4. The array will begin the Reconstruction process automatically. This can take a very long time depending on the size of your drives.
5. In this situation, original SPARE has now become AVAIL. So, re-add this AVAIL disk as SPARE using above steps. After setting this as spare – this disk will now show a status of: VDISKSP or SPARE

Before State of disks – one disk stuck in LEFTOVER while original SPARE stuck as AVAIL state in enclosure#2 (Tier 2 Storage – RAID5 is now critical):

After performing this rebuilding vdisk after a disk stuck/failed in LEFTOVER state (metadata cleared):

Notice that the disks are now marked as Vdisk under Reconstruction. See RAID rebuild progress below:
# show vdisks

Name            Size     Free    Own Pref   RAID   Disks Spr Chk  Status Jobs      Serial Number                    Drive Spin Down       Spin Down Delay   

------------------------------------------------------------------------------------------------------------------------------------------

ESX Prod Clstr  5995.1GB 0B      B   B      RAID5  11    1   64k  FTOL             00c0ffdb2868000084498b4d00000000 Disabled  0

Tier 2 Storage  9991.8GB 775.8GB A   A      RAID5  11    1   64k  CRIT   RCON 2%   00c0ffdb2f970000c84a924c00000000 Disabled  0

------------------------------------------------------------------------------------------------------------------------------------------

Success: Command completed successfully.

# show vdisks

Name            Size     Free    Own Pref   RAID   Disks Spr Chk  Status Jobs      Serial Number                    Drive Spin Down       Spin Down Delay   

------------------------------------------------------------------------------------------------------------------------------------------

ESX Prod Clstr  5995.1GB 0B      B   B      RAID5  11    1   64k  FTOL             00c0ffdb2868000084498b4d00000000 Disabled  0

Tier 2 Storage  9991.8GB 775.8GB A   A      RAID5  11    1   64k  CRIT   RCON 19%   00c0ffdb2f970000c84a924c00000000 Disabled  0

After Vdisk/RAID-5 reconstruction finishes, VDISK status shows normal/FTOL:

Events occurred as these process steps performed, Explained in details:

[www_tank]

Thank you for your detail reply!

Disk is absolutely new. Power-hiccup didn't happend because big APC UPS Symetra protects all equipment.
Other devices didn't notice any power-hiccup.

Below there is logs sequence:
A4001 Code8   A disk that is part of a vdisk is down.
A4002 Code314 There is a problem with a FRU.
A4003 Code1   A vdisk is critical
A4004 Code207 Vdisk scrub job failed
A4006 COde1   A vdisk is critical
A4007 Code9   A spare disk kicked in for a critical disk
A4008 Code37  Vdisk reconstruct started
A4012 Code18  Vdisk reconstruct completed successfully

Now:
Vdisk, Volumes, and hosts are still accessing due spare disk. Former spare disk got a VDISK state.
Everything (physical enclosure, volumes, vdisk) are in OK except one phisical disk.
Failed disk is in LEFTOVR state.

You would say me: There is nothing to worry about!
I am worry about that LEFTOVR disk doesn't work.
And there is no any spare disk in the vdisk now.
If that situation repeat again, vdisk won't be repaired and data may be lost.

Should I do "Clear meta data" procedure with LEFTOVR disk?
And after that Re-join physical disk to vdisk?

P.S. I know I have to practice spelling English 🙂 But Is there way to get MSA's log as a text with a web-interface?
I had to retype it manualy.

[DCESStorage]

I wouldn't kick-in LEFTOVR disk back into Vdisk configuration without checking if the disk itself has failed.

I am no MSA expert - but looknig at the " Vdisk scrub job failed" error, it sounds to me that particular LEFTOVR disk has most likely a problem disk. Have you talked to HP Tech support about doing detailed diagnosis and doing a disk replacement if possible ? Check on HP chat to see if your equipment is covered under warranty or contract and start a dialog with them. You need to capture Midplan serial# -- thats what contract/support keyed-in on.

Thanks

[qaiserkhan]

Hi;

We have RAID 50 configured with two global sphares on HP MSA, but one of the Vdisk get leftover status again and again and the reconstruct operation starts automatically. we have to clear meta data and have to add the leftover vdisk to global sphare again and again but it continuously happening from last few days.

if any one has clue/ reason why this happeing and what is the fix will be highly appreciated.

Thanks;

qaiser khan

[wwwtank]

This post can be checked "replied".

The disk was broken

[Sam10]

After a power failure one VDisk went to offline. 5 Disks of the same Vdisk are showing OK on the 1st enclosure but 7 Disks of the same Vdisk on the 2nd enclosure are in the state of LEFTOVER disks. Any ideas how to recover the data because I think that clearing the metadata on all 7 disks will lead to a data loss 

[aboubacar]

hello,

I've same issue and I've done same action.

unfortunately some data are not found when I mount the vdisk in RHEL5 system.

Can you help me.

Regards,

Aboubacar