After that I tried to delete other snapshots but I kept getting errors and the VM entered a Saved-Critical state. This will happen when Hyper-V Manager cannot access a file system location or cannot find a file, for instance when a removable hard drive is pulled out.

Approximately 30 seconds later, Hyper-V thought that it regained access to the location:

However, I couldn’t get any snapshots to delete and the virtual machine wouldn’t start. After a few minutes of panicked clicking I decided to restart the Hyper-V services. When they came back up my VM disappeared. The Virtual Machine configuration file was corrupted.

The next event suggests that a snapshot file was also corrupted.

These 16310 and 16330 errors repeated for a while. Panic continued. Eventually I rebooted. On reboot the VM was still missing and the 16310/16330 errors persisted.

On a hunch I decided to see what the AVHD files (the differencing disks that correlate with snapshot states) looked like.

This looked very much like what I would have expected if nothing had gone wrong (and if none of the snapshot deletions completed). Sticking with this line of inquiry (and what the 16310 error suggests), I created a new virtual machine and pointed it at the most recent AVHD file (selected above). All of my snapshots were missing but the virtual machine created successfully. I started the virtual machine and it was clearly in the same state it was in before I took the most recent snapshot, with a few caveats. In my panic I forgot to re-create my second NIC, so the VM started with only one (the one that I specified when I created the new VM). I also forgot to give it a second CPU. So I shut down the VM, made these changes, restarted, reconfigured the second NIC and tested that everything worked to my expectations. Recovery complete, so I shut down both VMs again.

At this point I’d recovered the VM but I still had a bunch of unnecessary data in my branch of differencing disks. In order to clean this up, I took a new snapshot of both VMs and exported the latest snapshot of each of them. This merged all the differences across the AVHD files in to a new, self-contained VHD file. After the exports finished I deleted the old VMs, waited for the Destroy operations to complete, cleaned up lingering files on the file system and imported the new exports. I took a new snapshot, as this is my new stable starting point and everything was (relatively speaking) back to normal. Phew!

With hindsight, I would have handled the recovery as follows:

• Create new VM, pointing at latest differencing disk (or whichever snapshot state you wanted to preserve).
• Reconfigure processors.
• Reconfigure network adapters in Hyper-V Manager.
• Start the virtual machine.
• Reconfigure NICs in the guest.
• Reboot.
• Test everything is working as expected in the VM.
• Shut Down.
• Snapshot.
• Export.
• Delete old VM from Hyper-V Manager.
• Wait for the Destroy operation to complete.
• Delete any lingering files from the file system.
• Import the exported virtual machines.

As I hinted at above, having gone through this process, it occurred to me that you could probably point at whichever AVHD file you wanted to, if you didn’t want to use the latest snapshot, assuming none of the AVHD files were corrupted. In this case it was just the virtual machine XML file and possibly the snapshot file that were corrupted, rather than the VHD file and differencing disks (AVHD files) themselves. The problem would be identifying which AVHD file corresponds to the snapshot that you want to keep, but in principal I think this would work.

I should note that this is probably unsupported, but you’re not really losing anything because otherwise you would have only been able to recover the first VHD file. This technique wouldn’t be much use if you didn’t know which snapshot you were after or if you wanted to recover the entire snapshot tree, but this fix gives you some recovery where the virtual machine file and the snapshot tree are corrupted but the disk data is not.

First, I found it was necessary to disable the DNS cache on the Hyper-V server so that it would find the newly-registered VPN-connected IP address of the client machine so that it can correctly route all traffic back to it. This can be rectified for testing purposes with an IPCONFIG /FLUSHNDS but it would be a massive annoyance if all remote Hyper-V users had to log on to the Hyper-V server via remote desktop and flush the DNS cache before connecting with Hyper-V Manager. In order to disable DNS caching at all times just disable the DNS client service. Granted, this introduces the overhead of DNS requests for all network activity on the root partition, but in this case we can live with that, especially since most Hyper-V hosts probably won’t have a lot of traffic on the management NIC.

Once this fix was in place we could establish a Hyper-V Manager connection over VPN, the server’s activity pane expanded and I could even modify the server’s settings. Strangely, the Hyper-V virtual machine pane remained empty and never enumerated the guests. At this point I started to detail the new problem for John and he suggested connecting using the WMI testing tool, WBEMTEST. He provided me with the query syntax and I confirmed that WMI was able to enumerate all of the virtual machines over VPN when Hyper-V Manager could not. John also asked if we were using SC Virtual Machine Manager (VMM) in this environment (we are), which unfortunately means that we can’t use his HVRemote tool to gather additional test results, as it doesn’t support VMM-managed Hyper-V servers.

We’re going to rebuild the Hyper-V Server on Windows Server 2008 R2 RTM in a few weeks and test connections before bringing it under VMM-management and test with HVRemote in more detail if there are problems. Until then, it would be interesting to know if anyone else has this problem with VMM-managed Hyper-V servers.

22/10/09 update: we still have the same problem after rebuilding the server. The rebuilt machine is not VMM-manager. Still no solution for this presently.