I had a bit of a storage outage in my lab due to a funky behaviour on the Synology that I use as primary storage for all my VMs:
Today I Learned: Adding IP address to interfaces on Synology causes reboots :/ Lab now in not such great shape. 80 VMs with APD. #vExpert
— Myles Gray (@mylesagray) November 13, 2016
Most stuff came back up or could at least be trivially fixed (like VCSA, PSCs, etc) you can edit the GRUB boot string and force into /bin/bash then run fsck from there.
One VM that doesn’t allow the GRUB string to be edited or both to be paused in any way is the NSX manager, that was a problem given I was presented with this upon boot:
/dev/sda2: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
This is not a good situation, a corrupt FS and no way to force into a shell.
A quick cry for advice on the vExpert Slack and it was decided booting into a live CD and trying to mount and repair the filesystems from there was the way to go.
So I downloaded the Ubuntu Desktop installer ISO (the server installer does not have fsck present) – transferred the ISO to my NFS datastore and mounted to the NSX Manager VM. If you’re going to do this, make sure you force BIOS boot mode on the VM and put boot from CD-ROM at the top of the list.
Next, reboot into the Ubuntu ISO and choose “Try Ubuntu” when presented with the option to try or install.
Once the live environment is up – run Terminal and then type ls -l /dev/sd* to list all discovered partitions.
Then you run sudo fsck /dev/sda2 and hit y when prompted to fix any errors found on the filesystem. At this stage your filesystem should be “clean”, so if you run sudo fsck /dev/sda2 again it should show up as so:
Unmount the ISO from the VM, change the boot order back to normal and you should be good to go again:
Leave a Reply