I have come across a number of environments where mystery “snapshot” files exist - they are not seen in snapshot manager, running a consolidation them doesn’t help, creating a snapshot (with memory, or guest OS quiescing) then running “Delete All” doesn’t resolve it, but some applications still think a snapshot is there.
To take care of these is quite a manual process after you have followed all the VMware KB advice:
So we’ve done all the above, don’t fancy doing a V2V to sort this as it shouldn’t really be a problem in the first place.
First step is to find all snapshots and delta disks on the datastores:
# find /vmfs/volumes/ -name *-delta*;find /vmfs/volumes/ -name *-0000*
/vmfs/volumes/[id]/[VM]/VM-000002-delta.vmdk
/vmfs/volumes/[id]/[VM]/VM-000002.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002-ctk.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002-delta.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002.vmdk
As you can see above there are 5 snapshot/tracking/vmdk files that are orphaned and we need to investigate.
The first step is to snapshot the VM above and then run delete all to see if VMware can clear them all down - re-run the check above, if they still exist it is quite possible they are orphaned.
To investigate further we can find out what vmdks are mounted by the vmx for that particular VM:
# cat /vmfs/volumes/[id]/[VM]/VM.vmx | grep vmdk
scsi0:0.fileName = "VM.vmdk"
scsi0:1.fileName = "VM_1.vmdk"
So this gives us all the vmdks mounted by the VM - we can then cat these files to check what underlying files on the datastore they reference (I have done one of the two disks as an example):
# cat VM_1.vmdk
# Disk DescriptorFile
version=3
encoding="UTF-8"
CID=IDHERE
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"
# Extent description
RW 220200960 VMFS "VM_1-flat.vmdk"
# Change Tracking File
changeTrackPath="VM_1-ctk.vmdk"
We are interested in the two sections from above “Extent description” and “Change Tracking File”, from the above we can see the reference VMDKs files are:
VM_1-flat.vmdk
VM_1-ctk.vmdk
In the interests of completeness a cat of the other VMDK (VM.vmdk) showed the following were referenced:
VM-flat.vmdk
VM-ctk.vmdk
Check if the vmdk files are locked by any hosts, -delta, -ctk and -flat files should be locked when in active I/O use, descriptor files (just the .vmdk “meta” files) are not so you need to check all the files individually:
# vmkfstools -D VM-000002-delta.vmdk
Lock [type 10c00001 offset 39821312 v 23225, hb offset 3702784
gen 7489, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2048078
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 66, 20>, gen 27, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 606, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536
If it was locked by an ESXi host the MAC of the host would be shown in the owner readout above - all zeros indicates no R/W lock. From the VMware docs:
If the command
vmkfstools -D VM-000002-delta.vmdkdoes not return a valid MAC address in the top field (returns all zeros ). Review the field below it, theRO Ownerline below it to see which MAC address owns the read only/multi writer lock on the file.In some cases it is possible that it is a Service Console-based lock, an NFS lock or a lock generated by another system or product that can use or read VMFS file systems. The file is locked by a VMkernel child or cartel world and the offending host running the process/world must be rebooted to clear it.
Once you have identified the host or backup tool (machine that owns the MAC) locking the file, power it off or stop the responsible service, then restart the management agents on the host running the virtual machine to release the lock.
So no references to our 5 mystery files - check the last time they were used by running:
# ls -ltr /vmfs/volumes/[id]/[VM]/ | grep vmdk
-rw------- 1 root root 16863232 Nov 13 15:01 VM-000002-delta.vmdk
-rw------- 1 root root 344 Nov 13 15:01 VM-000002.vmdk
-rw------- 1 root root 6554112 Nov 13 15:01 VM_1-000002-ctk.vmdk
-rw------- 1 root root 16986112 Nov 13 15:01 VM_1-000002-delta.vmdk
-rw------- 1 root root 419 Nov 13 15:01 VM_1-000002.vmdk
-rw------- 1 root root 2621952 Feb 5 22:01 VM-ctk.vmdk
-rw------- 1 root root 612 Feb 5 22:01 VM_1.vmdk
-rw------- 1 root root 606 Feb 5 22:01 VM.vmdk
-rw------- 1 root root 6881792 Feb 5 22:01 VM_1-ctk.vmdk
-rw------- 1 root root 42949672960 Feb 6 15:20 VM-flat.vmdk
-rw------- 1 root root 112742891520 Feb 6 15:20 VM_1-flat.vmdk
As we can see above our orphaned files were last accessed almost 3 months previous.
Then make sure they are not locked by a process, touch them and see that the timestamp updates:
# touch /vmfs/volumes/[id]/[VM]/*-00000* | ls -ltr | grep vmdk
-rw------- 1 root root 612 Feb 5 22:01 VM_1.vmdk
-rw------- 1 root root 606 Feb 5 22:01 VM.vmdk
-rw------- 1 root root 6881792 Feb 5 22:01 VM_1-ctk.vmdk
-rw------- 1 root root 2621952 Feb 5 22:01 VM-ctk.vmdk
-rw------- 1 root root 42949672960 Feb 6 15:29 VM-flat.vmdk
-rw------- 1 root root 419 Feb 6 15:29 VM_1-000002.vmdk
-rw------- 1 root root 16986112 Feb 6 15:29 VM_1-000002-delta.vmdk
-rw------- 1 root root 6554112 Feb 6 15:29 VM_1-000002-ctk.vmdk
-rw------- 1 root root 112742891520 Feb 6 15:29 VM_1-flat.vmdk
-rw------- 1 root root 344 Feb 6 15:29 VM-000002.vmdk
-rw------- 1 root root 16863232 Feb 6 15:29 VM-000002-delta.vmdk
Being able to touch the file, run vmkfstools -D finding no locks, find no references in vmdk descriptor files generally means it isn’t in active use and is safe to move/remove, create a new create a new directory and move the suspect files to it and check for problems with the VM:
# mkdir oldfiles
# mv *-00000* oldfiles/.
If all looks well and you are happy the VM is operating normally delete the directory:
# rm -r oldfiles/
References:
Why not follow @mylesagray on Twitter ↗ for more like this!