Blah, Cloud.

Adventures in architectures

  • Twitter
  • GitHub
  • Home
  • Blog
  • Kubernetes on vSphere
  • Multi-tenant IaaS Networking
  • Me
    • About
    • CV
    • Contact
Home » Blog » Infrastructure » Safely check/remove orphaned VMDK files from ESXi

Safely check/remove orphaned VMDK files from ESXi

21/02/2015 by Myles Gray 8 Comments

I have come across a number of environments where mystery “snapshot” files exist – they are not seen in snapshot manager, running a consolidation them doesn’t help, creating a snapshot (with memory, or guest OS quiescing) then running “Delete All” doesn’t resolve it, but some applications still think a snapshot is there.

To take care of these is quite a manual process after you have followed all the VMware KB advice:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2017072&src=vmw_so_vex_mgray_1080 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1023657&src=vmw_so_vex_mgray_1080 http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1002310&src=vmw_so_vex_mgray_1080

So we’ve done all the above, don’t fancy doing a V2V to sort this as it shouldn’t really be a problem in the first place.

First step is to find all snapshots and delta disks on the datastores:

# find /vmfs/volumes/ -name *-delta*;find /vmfs/volumes/ -name *-0000*
/vmfs/volumes/[id]/[VM]/VM-000002-delta.vmdk
/vmfs/volumes/[id]/[VM]/VM-000002.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002-ctk.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002-delta.vmdk
/vmfs/volumes/[id]/[VM]/VM_1-000002.vmdk

As you can see above there are 5 snapshot/tracking/vmdk files that are orphaned and we need to investigate.

The first step is to snapshot the VM above and then run delete all to see if VMware can clear them all down – re-run the check above, if they still exist it is quite possible they are orphaned.

To investigate further we can find out what vmdks are mounted by the vmx for that particular VM:

# cat /vmfs/volumes/[id]/[VM]/VM.vmx | grep vmdk
scsi0:0.fileName = "VM.vmdk"
scsi0:1.fileName = "VM_1.vmdk"

So this gives us all the vmdks mounted by the VM – we can then cat these files to check what underlying files on the datastore they reference (I have done one of the two disks as an example):

# cat VM_1.vmdk
# Disk DescriptorFile
version=3
encoding="UTF-8"
CID=IDHERE
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 220200960 VMFS "VM_1-flat.vmdk"

# Change Tracking File
changeTrackPath="VM_1-ctk.vmdk"

We are interested in the two sections from above “Extent description” and “Change Tracking File”, from the above we can see the reference VMDKs files are:

VM_1-flat.vmdk
VM_1-ctk.vmdk

In the interests of completeness a cat of the other VMDK (VM.vmdk) showed the following were referenced:

VM-flat.vmdk
VM-ctk.vmdk

Check if the vmdk files are locked by any hosts, -delta, -ctk and -flat files should be locked when in active I/O use, descriptor files (just the .vmdk “meta” files) are not so you need to check all the files individually:

# vmkfstools -D VM-000002-delta.vmdk
Lock [type 10c00001 offset 39821312 v 23225, hb offset 3702784
gen 7489, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2048078
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 66, 20>, gen 27, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 606, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 65536

If it was locked by an ESXi host the MAC of the host would be shown in the owner readout above – all zeros indicates no R/W lock. From the VMware docs:

If the command “vmkfstools -D VM-000002-delta.vmdk ” does not return a valid MAC address in the top field (returns all zeros ). Review the field below it, the RO Owner line below it to see which MAC address owns the read only/multi writer lock on the file.

In some cases it is possible that it is a Service Console-based lock, an NFS lock or a lock generated by another system or product that can use or read VMFS file systems. The file is locked by a VMkernel child or cartel world and the offending host running the process/world must be rebooted to clear it.

Once you have identified the host or backup tool (machine that owns the MAC) locking the file, power it off or stop the responsible service, then restart the management agents on the host running the virtual machine to release the lock.

So no references to our 5 mystery files – check the last time they were used by running:

# ls -ltr /vmfs/volumes/[id]/[VM]/ | grep vmdk
-rw-------    1 root     root      16863232 Nov  13 15:01 VM-000002-delta.vmdk
-rw-------    1 root     root           344 Nov  13 15:01 VM-000002.vmdk
-rw-------    1 root     root       6554112 Nov  13 15:01 VM_1-000002-ctk.vmdk
-rw-------    1 root     root      16986112 Nov  13 15:01 VM_1-000002-delta.vmdk
-rw-------    1 root     root           419 Nov  13 15:01 VM_1-000002.vmdk
-rw-------    1 root     root       2621952 Feb  5 22:01 VM-ctk.vmdk
-rw-------    1 root     root           612 Feb  5 22:01 VM_1.vmdk
-rw-------    1 root     root           606 Feb  5 22:01 VM.vmdk
-rw-------    1 root     root       6881792 Feb  5 22:01 VM_1-ctk.vmdk
-rw-------    1 root     root     42949672960 Feb  6 15:20 VM-flat.vmdk
-rw-------    1 root     root     112742891520 Feb  6 15:20 VM_1-flat.vmdk

As we can see above our orphaned files were last accessed almost 3 months previous.

Then make sure they are not locked by a process, touch them and see that the timestamp updates:

# touch /vmfs/volumes/[id]/[VM]/*-00000* | ls -ltr | grep vmdk
-rw-------    1 root     root           612 Feb  5 22:01 VM_1.vmdk
-rw-------    1 root     root           606 Feb  5 22:01 VM.vmdk
-rw-------    1 root     root       6881792 Feb  5 22:01 VM_1-ctk.vmdk
-rw-------    1 root     root       2621952 Feb  5 22:01 VM-ctk.vmdk
-rw-------    1 root     root     42949672960 Feb  6 15:29 VM-flat.vmdk
-rw-------    1 root     root           419 Feb  6 15:29 VM_1-000002.vmdk
-rw-------    1 root     root      16986112 Feb  6 15:29 VM_1-000002-delta.vmdk
-rw-------    1 root     root       6554112 Feb  6 15:29 VM_1-000002-ctk.vmdk
-rw-------    1 root     root     112742891520 Feb  6 15:29 VM_1-flat.vmdk
-rw-------    1 root     root           344 Feb  6 15:29 VM-000002.vmdk
-rw-------    1 root     root      16863232 Feb  6 15:29 VM-000002-delta.vmdk

Being able to touch the file, run vmkfstools -D finding no locks, find no references in vmdk descriptor files generally means it isn’t in active use and is safe to move/remove, create a new create a new directory and move the suspect files to it and check for problems with the VM:

# mkdir oldfiles
# mv *-00000* oldfiles/.

If all looks well and you are happy the VM is operating normally delete the directory:

# rm -r oldfiles/

References:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051&src=vmw_so_vex_mgray_1080 http://blogs.vmware.com/vsphere/2012/08/some-useful-vmkfstools-hidden-options.html?src=vmw_so_vex_mgray_1080

Why not follow @mylesagray on Twitter for more like this!

Show some love:

  • Reddit
  • Twitter
  • Pocket
  • LinkedIn
  • Email
  • Telegram

Similar things I've written

Filed Under: Infrastructure, Virtualisation Tagged With: esxi, orphaned, vmdk, vmfs, vmkfstools, vmware, vsphere

About Myles Gray

Hi! I'm Myles, and I'm a Dev Advocate at VMware. Focused primarily on content generation, product enablement and feedback from customers and field to engineering.

Comments

  1. Brian says

    01/12/2015 at 17:12

    Very good article. I remember looking for a guide like this last year. I will definitely go over your steps when I have this issue again.

    Reply
  2. sebastiangrugel says

    06/05/2016 at 19:52

    Good article.

    Reply
  3. Mubara says

    03/02/2017 at 17:00

    Awesome Article !

    Reply
  4. Ptochos says

    18/10/2017 at 19:22

    Outstanding article – Thanks so much!
    I really appreciate articles like this that logically step through the process, and explain reasoning behind the steps.

    Reply
  5. Ehsan says

    19/03/2018 at 04:58

    This article saved my life…

    Reply
  6. Kevin Greenway says

    04/05/2019 at 07:05

    Excellent article, thank you very much for taking the time to write this!

    Reply
  7. aungangster says

    03/06/2019 at 04:31

    Good article , I was looking for this for a while.
    Thank you very much

    Reply
  8. Roman Revell (@RoTwit) says

    28/03/2020 at 03:57

    Wow. All that and still it’s “generally” ok and we have to hold on to the files for a while to make sure the machine does not blow up. That’s too much uncertainty when I have like 20 files per machine and several machines (potentially tens of machines) that have this problem. This makes me not want to use VMware. Or at least not be the one to maintain VMs. I definitely need an automated cleanup utility instead of doing all of this. But, thanks. This might be the only way.

    Reply

Leave a Reply Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Myles Gray

Hi! I'm Myles, and I'm a Dev Advocate at VMware. Focused primarily on content generation, product enablement and feedback from customers and field to engineering. Read More…

Categories

Tags

active directory authentication CBT cisco datastore dell design esxi fortigate iscsi jumbo frame kubernetes lab linux load-balancing lun md3000i mtu networking NginX nic nsx openSUSE osx pxe readynas san sdelete serial teaming ubuntu vcenter vcloud director vcsa vexpert video VIRL vmdk vmfs vmware vsan vsphere vsphere 6 vsphere beta windows

Subscribe to Blog via Email

Copyright © 2021 · News Pro Theme on Genesis Framework · WordPress · Log in

loading Cancel
Post was not sent - check your email addresses!
Email check failed, please try again
Sorry, your blog cannot share posts by email.