Blah, Cloud.

Adventures in architectures

  • Twitter
  • GitHub
  • Home
  • Blog
  • Kubernetes on vSphere
  • Multi-tenant IaaS Networking
  • Me
    • About
    • CV
    • Contact
Home » Blog » Virtualisation » Recovering NSX Manager with corrupt filesystem

Recovering NSX Manager with corrupt filesystem

13/11/2016 by Myles Gray 1 Comment

I had a bit of a storage outage in my lab due to a funky behaviour on the Synology that I use as primary storage for all my VMs:

Today I Learned: Adding IP address to interfaces on Synology causes reboots :/ Lab now in not such great shape. 80 VMs with APD. #vExpert

— Myles Gray (@mylesagray) November 13, 2016

Most stuff came back up or could at least be trivially fixed (like VCSA, PSCs, etc) you can edit the GRUB boot string and force into /bin/bash then run fsck from there.

One VM that doesn’t allow the GRUB string to be edited or both to be paused in any way is the NSX manager, that was a problem given I was presented with this upon boot:

NSX Manager No Boot

/dev/sda2: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

This is not a good situation, a corrupt FS and no way to force into a shell.

A quick cry for advice on the vExpert Slack and it was decided booting into a live CD and trying to mount and repair the filesystems from there was the way to go.

So I downloaded the Ubuntu Desktop installer ISO (the server installer does not have fsck present) – transferred the ISO to my NFS datastore and mounted to the NSX Manager VM. If you’re going to do this, make sure you force BIOS boot mode on the VM and put boot from CD-ROM at the top of the list.

Force into BIOS mode on boot

Next, reboot into the Ubuntu ISO and choose “Try Ubuntu” when presented with the option to try or install.

Once the live environment is up – run Terminal and then type ls -l /dev/sd* to list all discovered partitions.

List partitions discovered

Then you run sudo fsck /dev/sda2 and hit y when prompted to fix any errors found on the filesystem. At this stage your filesystem should be “clean”, so if you run sudo fsck /dev/sda2 again it should show up as so:

SDA2 clean filesystem

Unmount the ISO from the VM, change the boot order back to normal and you should be good to go again:

NSX Manager working again

Why not follow @mylesagray on Twitter for more like this!

Show some love:

  • Reddit
  • Twitter
  • Pocket
  • LinkedIn
  • Email
  • Telegram

Similar things I've written

Filed Under: Infrastructure, Networks, Virtualisation Tagged With: fsck, linux, nsx, vmware

About Myles Gray

Hi! I'm Myles, and I'm a Dev Advocate at VMware. Focused primarily on content generation, product enablement and feedback from customers and field to engineering.

Comments

  1. Subs says

    27/08/2017 at 18:10

    Godsent article.!

    Reply

Leave a Reply Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Myles Gray

Hi! I'm Myles, and I'm a Dev Advocate at VMware. Focused primarily on content generation, product enablement and feedback from customers and field to engineering. Read More…

Categories

Tags

active directory authentication CBT cisco datastore dell design esxi fortigate iscsi jumbo frame kubernetes lab linux load-balancing lun md3000i mtu networking NginX nic nsx openSUSE osx pxe readynas san sdelete serial teaming ubuntu vcenter vcloud director vcsa vexpert video VIRL vmdk vmfs vmware vsan vsphere vsphere 6 vsphere beta windows

Subscribe to Blog via Email

Copyright © 2021 · News Pro Theme on Genesis Framework · WordPress · Log in

loading Cancel
Post was not sent - check your email addresses!
Email check failed, please try again
Sorry, your blog cannot share posts by email.