Safely check/remove orphaned VMDK files from ESXi

I have come across a number of environments where mystery “snapshot” files exist – they are not seen in snapshot manager, running a consolidation them doesn’t help, creating a snapshot (with memory, or guest OS quiescing) then running “Delete All” doesn’t resolve it, but some applications still think a snapshot is there.

To take care of these is quite a manual process after you have followed all the VMware KB advice:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2017072&src=vmw_so_vex_mgray_1080 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1023657&src=vmw_so_vex_mgray_1080 http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1002310&src=vmw_so_vex_mgray_1080

So we’ve done all the above, don’t fancy doing a V2V to sort this as it shouldn’t really be a problem in the first place.

First step is to find all snapshots and delta disks on the datastores:

# find /vmfs/volumes/ -name *-delta*;find /vmfs/volumes/ -name *-0000*
/vmfs/volumes/[id]/[VM]/VM-000002-delta.vmdk
/vmfs/volumes/[id]/[VM]/VM-000002.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002-ctk.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002-delta.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002.vmdk

As you can see above there are 5 snapshot/tracking/vmdk files that are orphaned and we need to investigate.

The first step is to snapshot the VM above and then run delete all to see if VMware can clear them all down – re-run the check above, if they still exist it is quite possible they are orphaned.

To investigate further we can find out what vmdks are mounted by the vmx for that particular VM:

# cat /vmfs/volumes/[id]/[VM]/VM.vmx | grep vmdk
scsi0:0.fileName = "VM.vmdk"
scsi0:1.fileName = "VM_1.vmdk"

So this gives us all the vmdks mounted by the VM – we can then cat these files to check what underlying files on the datastore they reference (I have done one of the two disks as an example):

# cat VM_1.vmdk
# Disk DescriptorFile
version=3
encoding="UTF-8"
CID=IDHERE
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 220200960 VMFS "VM_1-flat.vmdk"

# Change Tracking File
changeTrackPath="VM_1-ctk.vmdk"

We are interested in the two sections from above “Extent description” and “Change Tracking File”, from the above we can see the reference VMDKs files are:

VM_1-flat.vmdk
VM_1-ctk.vmdk

In the interests of completeness a cat of the other VMDK (VM.vmdk) showed the following were referenced:

VM-flat.vmdk
VM-ctk.vmdk

Check if the vmdk files are locked by any hosts, -delta, -ctk and -flat files should be locked when in active I/O use, descriptor files (just the .vmdk “meta” files) are not so you need to check all the files individually:

# vmkfstools -D VM-000002-delta.vmdk
Lock [type 10c00001 offset 39821312 v 23225, hb offset 3702784
gen 7489, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2048078
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 66, 20>, gen 27, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 606, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536

If it was locked by an ESXi host the MAC of the host would be shown in the owner readout above – all zeros indicates no R/W lock. From the VMware docs:

If the command “vmkfstools -D VM-000002-delta.vmdk ” does not return a valid MAC address in the top field (returns all zeros ). Review the field below it, the RO Owner line below it to see which MAC address owns the read only/multi writer lock on the file.

In some cases it is possible that it is a Service Console-based lock, an NFS lock or a lock generated by another system or product that can use or read VMFS file systems. The file is locked by a VMkernel child or cartel world and the offending host running the process/world must be rebooted to clear it.

Once you have identified the host or backup tool (machine that owns the MAC) locking the file, power it off or stop the responsible service, then restart the management agents on the host running the virtual machine to release the lock.

So no references to our 5 mystery files – check the last time they were used by running:

# ls -ltr /vmfs/volumes/[id]/[VM]/ | grep vmdk
-rw-------    1 root     root      16863232 Nov  13 15:01 VM-000002-delta.vmdk
-rw-------    1 root     root           344 Nov  13 15:01 VM-000002.vmdk
-rw-------    1 root     root       6554112 Nov  13 15:01 VM_1-000002-ctk.vmdk
-rw-------    1 root     root      16986112 Nov  13 15:01 VM_1-000002-delta.vmdk
-rw-------    1 root     root           419 Nov  13 15:01 VM_1-000002.vmdk
-rw-------    1 root     root       2621952 Feb  5 22:01 VM-ctk.vmdk
-rw-------    1 root     root           612 Feb  5 22:01 VM_1.vmdk
-rw-------    1 root     root           606 Feb  5 22:01 VM.vmdk
-rw-------    1 root     root       6881792 Feb  5 22:01 VM_1-ctk.vmdk
-rw-------    1 root     root     42949672960 Feb  6 15:20 VM-flat.vmdk
-rw-------    1 root     root     112742891520 Feb  6 15:20 VM_1-flat.vmdk

As we can see above our orphaned files were last accessed almost 3 months previous.

Then make sure they are not locked by a process, touch them and see that the timestamp updates:

# touch /vmfs/volumes/[id]/[VM]/*-00000* | ls -ltr | grep vmdk
-rw-------    1 root     root           612 Feb  5 22:01 VM_1.vmdk
-rw-------    1 root     root           606 Feb  5 22:01 VM.vmdk
-rw-------    1 root     root       6881792 Feb  5 22:01 VM_1-ctk.vmdk
-rw-------    1 root     root       2621952 Feb  5 22:01 VM-ctk.vmdk
-rw-------    1 root     root     42949672960 Feb  6 15:29 VM-flat.vmdk
-rw-------    1 root     root           419 Feb  6 15:29 VM_1-000002.vmdk
-rw-------    1 root     root      16986112 Feb  6 15:29 VM_1-000002-delta.vmdk
-rw-------    1 root     root       6554112 Feb  6 15:29 VM_1-000002-ctk.vmdk
-rw-------    1 root     root     112742891520 Feb  6 15:29 VM_1-flat.vmdk
-rw-------    1 root     root           344 Feb  6 15:29 VM-000002.vmdk
-rw-------    1 root     root      16863232 Feb  6 15:29 VM-000002-delta.vmdk

Being able to touch the file, run vmkfstools -D finding no locks, find no references in vmdk descriptor files generally means it isn’t in active use and is safe to move/remove, create a new create a new directory and move the suspect files to it and check for problems with the VM:

# mkdir oldfiles
# mv *-00000* oldfiles/.

If all looks well and you are happy the VM is operating normally delete the directory:

# rm -r oldfiles/

References:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051&src=vmw_so_vex_mgray_1080 http://blogs.vmware.com/vsphere/2012/08/some-useful-vmkfstools-hidden-options.html?src=vmw_so_vex_mgray_1080

Why not follow @mylesagray on Twitter for more like this!