My VMware ESX server console volume went readonly. How can I save my VMs?

Question :

My VMware ESX server console volume went readonly. How can I save my VMs?,

Answer :

Two RAID volumes, VMware kernel/console running on a RAID1, vmdks live on a RAID5. Entering a login at the console just results in SCSI errors, no password prompt. Praise be, the VMs are actually still running. We’re thinking, though, that upon reboot the kernel may not start again and the VMs will be down.

We have database and disk backups of the VMs, but not backups of the vmdks themselves.

What are my options?

Our current best idea is

  1. Use VMware Converter to create live vmdks from the running VMs, as if it was a P2V migration.
  2. Reboot host server and run RAID diagnostics, figure out what in the “h” happened
  3. Attempt to start ESX again, possibly after rebuilding its RAID volume
  4. Possibly have to re-install ESX on its volume and re-attach VMs
  5. If that doesn’t work, attach the “live” vmdks created in step 1 to a different VM host.
Read More  How do you balance fun feature creep with time constraints?

,

It was the backplane. Both drives of the RAID1 and one drive of the RAID5 were inaccessible. Incredibly, the VMware hypervisor continued to run for three days from memory with no access to its host disk, keeping the VMs it managed alive.

At step 3 above we diagnosed the hardware problem and replaced the RAID controller, cables, and backplane. After restart, we re-initialized the RAID by instructing the controller to query the drives for their configurations. Both were degraded and both were repaired successfully.

Read More  Are foreign keys really necessary in a database design?

At step 4, it was not necessary to reinstall ESX; although, at bootup, it did not want to register the VMs. We had to dig up some buried management stuff to instruct the kernel to resignature the VMs. (Search VM docs for “resignature.”)

I believe that our fallback plan would have worked, the VMware Converter images of the VMs that were running “orphaned” were tested and ran fine with no data loss. I highly recommend performing a VMware Converter imaging of any VM that gets into this state, after shutting down as many services as possible and getting the VM into as read-only a state as possible. Loading a vmdk either elsewhere or on the original host as a repair is usually going to be WAY faster than rebuilding a server from the ground up with backups.

Read More  What are the advantages/disadvantages of using a CTE?

That’s the answer My VMware ESX server console volume went readonly. How can I save my VMs?, Hope this helps those looking for an answer. Then we suggest to do a search for the next question and find the answer only on our site.

Disclaimer :

The answers provided above are only to be used to guide the learning process. The questions above are open-ended questions, meaning that many answers are not fixed as above. I hope this article can be useful, Thank you