Practical VMFS signatures

VMFS is a pretty cool (if relatively undocumented) filesystem, and VMFS volumes were designed with one particular quirk that’s both a blessing and a curse — when you create the volume, ESX writes a few pieces of information to the disk signature on the volume that helps it identify the volume and figure out what to do with it. Each volume contains a UUID used to uniquely identify it when multiple volumes with the same name are presented, but each volume also registers one of three unique identifiers to the volume — the Network Addressing Authority (NAA) ID, an Extended Unique Identifier (EUI), or, on storage subsystems that don’t support either of the other two identifiers, a LUN number. If you’re interested in the low-level nitty-gritty of it, Ubiquitous Talk published a really great blog entry on VMFS on-disk signatures that you should read.

This decision was made because most people run VMware off of Fibre Channel or iSCSI SAN, where you may do something like taking a copy-on-write snapshot of a VMFS volume and presenting it back to the host as a read-only volume. ESX compares the identifier presented by the LUN to the one written to the disk signature, the one that it expects to see. If they mismatch, it’s assumed that it’s not the original LUN and that it’s a copy. Sometimes this isn’t the case, and your underlying storage has actually changed, either because you made a copy to another LUN or because you’re trying to mount a replicated copy on another SAN at your disaster recovery site. In these cases, to mount the volume as a normal writable volume, you need to resignature it, which re-writes all of the above information to the disk. Since this information includes the UUID, and ESX uses that UUID to reference virtual machines in its inventory, you’ll need to manually re-add all of the virtual machines back to your cluster. This is one of those annoying things that Site Recovery Manager was designed to automate (see TechTarget article).

ESX 3.5 used to automatically mount and present the volume as a read-only snapshot with a new unique name. If you wanted to resignature the volume, you would set the advanced setting LVM.EnableResignature to 1, and you would rescan for volumes. This had the unfortunate consequence of re-signaturing all volumes, even if you only intended to resignature one of them. A new esxcfg-volume command was added to perform this operation, and VMware changed the default behavior in the GUI so that if a volume is detected as a snapshot, you have to manually add the storage, at which time you’ll be prompted about whether you want to mount the volume as a snapshot or if you want to resignature it.

Problem 1: Cloned boot LUNs don’t boot

As of ESX 4, the service console resides in a VMDK on a VMFS volume, so you can run into major issues if you boot from SAN and fail your boot LUNs over to your DR site, because the filesystem used to store your service console is subject to the exact same signaturing issues as other VMFS volumes. The boot LUNs on each server need to be manually resignatured, which is covered by Get VIRTical and VMware KB 1012874.

Problem 2: Non-contiguous extents don’t resignature (right now)

A couple of weeks ago, I ran into a rather significant and nasty regression in vSphere 4. I had taken a 500GB VMFS volume, added a second 500GB extent, and then I had grown each of the volumes by 250GB. As a result, the first LUN occupied 0-500GB and 1000-1250GB in the VMFS volume, while the second LUN occupied 500-1000GB and 1250-1500GB. Notice that the second disk’s start is before the first disk’s end (and this should be fine in any reasonable logical volume manager).

This worked fine for months. When I failed over to our DR site, the volume was detected as a snapshot and I couldn’t resignature because esxcfg-volume thought the LUNs overlapped:


[root@esx01 ~]# esxcfg-volume -l
VMFS3 UUID/label: 4bc639b4-21bbc059-d77b-e41f132c2a8a/shared-esxdev
Can mount: No (duplicate extents found)
Can resignature: No (duplicate extents found)
Extent name: naa.600a0b800047f5f20000bc934bf1480e:1 range: 0 - 1279487 (MB)
Extent name: naa.600a0b80006e09620000bc914bf14835:1 range: 511744 - 1535487 (MB)


After going back and forth with VMware for a very long time on this issue, they finally determined it to be a bug in 4.0 that prevents 4.0 from resignaturing the volume. Don’t extend any volumes defined as extents in a VMFS filesystem (VMware’s recommendation is to not use extents at all unless you absolutely need them to extend a VMFS volume beyond 2TB). If this issue bites you, you can get around it by presenting the volume to an ESX 3.5 host, setting LVM.EnableResignature to 1, rescanning/resignaturing, and then presenting the LUNs back to an ESX 4 host. This should hopefully be fixed by 4.1 U1.

As a final aside: it looks like the open-source VMFS driver has similar problems with non-contiguous extents (it throws back garbage data). I haven’t reported that as a bug yet.