Tristan Watkins on IT Infrastructure » MVSMP Archive

MAC duplication issues with captured VMs and WDS

Tristan Watkins — Fri, 12 Mar 2010 01:07:34 +0000

I’ve previously reported problems with MAC duplication on Hyper-V host external network connections on Windows Server 2008 R2, which I’ve never fully resolved, although we have been successfully working around the issue as detailed in the first link above.

A couple of weeks ago I was working simultaneously on my Windows Server 2008 R2 laptop with Hyper-V (the same laptop build that’s been previously mentioned) and a Windows 7 x64 build that I was using for testing, when I noticed severe but intermittent network problems on both machines. After a fair amount of head scratching, I noticed that the two laptops had duplicated MAC addresses. Blatantly that shouldn’t happen, as the whole point of a MAC address is to provide uniqueness. The most perplexing issue was that the addresses conflicted across two different operating systems. However, it happened. Both wired adapters on the two machines had the MAC address 00-21-9B-DC-8E-0B. I uninstalled the wired adapter on the Windows 7 machine and scanned for new hardware. When the device reinstalled the problem went away.

Since then I’ve been poking around a bit and I’ve found this old TechNet article from December 2007 in which Wes Miller says:

In addition to changing the SID and the machine name, you also need to change certain values that may be specific to the virtual computing technology you’re using. In particular, you need to change the MAC address (the unique ID for networking devices). Plus, many virtual applications also have their own unique identifier. Most store these in their own machine configuration files, so you’ll want to know how to manipulate those entries (and maintain their validity). Note that many virtualization products that support Pre-Boot Execution Environment (PXE) key the SMBIOS UUID based on their own unique ID—emphasizing the need to change this (or let the virtualization software change it for you, if supported) if you’re joining it to a domain; otherwise, managing WDS or RIS-client systems can become impossible (if GUIDs conflict). Most of the virtualization solutions I’ve worked with can have severe networking problems in the case of duplicate MAC addresses; so if you are not just moving a virtual machine, it’s very important that you change the MAC address if the virtualization software does not do it for you.

That’s by no means conclusive, but I’m not turning up much else. So if you’re encountering network problems with WDS images that began their life as a virtual machine, you may want to consider automating the un/re-installation of the NICs post-deployment or addressing that with WAIK somehow. Otherwise just make sure to do it manually before giving the machine to the recipient, or people will wind up with a rather frustrating network issue.

If we get around to doing any more testing of this or gather more evidence from internal deployments, I’ll post the findings here. In the mean time I thought I’d update the ongoing saga here.

More on routine loss of external network connectivity on Hyper-V hosts (not guests)

Tristan Watkins — Thu, 22 Oct 2009 23:13:19 +0000

Further to my post from a few months ago on the this topic (dating back to the RC build), I’ve seen this same problem a few more times on the RTM build of Windows Server 2008 R2. My suggested approach still fixes the problem and it doesn’t recur, but I’ve still not been able to pin down a cause and I can find no documentation on host machine MAC assignments anywhere. Search engines results are entirely focused on MAC pool duplication, which should be a completely distinct issue since the MAC address in question falls outside of Hyper-V’s MAC pool.

To help pin this down, I will add that this Hyper-V host machine has been deployed from a WDS image, but I initially discounted that as an issue because the Hyper-V role is not part of the image; it is added during the deployment as specified in an unattended installation file and the Hyper-v networks are created by script post-deployment. Since the Hyper-V host’s MAC address is distinct from the physical adapter’s address, I’ve been assuming that there must be an algorithm that generates the host’s virtual MAC address uniquely, but given that this problem seems to manifest itself soon after installation I’m leaving this possibility on the table.

I’ve noticed a fair amount of traffic for this topic. If you’re having the same problem and this suggestion fixes it, I’d be grateful to know if the Hyper-V system was deployed from an image, as that might help to narrow the troubleshooting effort.

Routine loss of connectivity on a Hyper-V host’s external connection

Tristan Watkins — Tue, 25 Aug 2009 19:00:47 +0000

We’ve recently been piloting a laptop developer build on Windows Server 2008 R2 Release Candidate (Build 7100) with the Hyper-V role. One of the first receipients of this build complained of connectivity problems in Office Communicator once every minute or two. For as-yet undiagnosed reasons we have lengthy sign-in times for Communicator, so this loss of connectivity rendered it completely unusable. This same problem was visible in Outlook, although less disruptive since we use Cached Exchange Mode. Both Exchange and the OCS server are hosted but we also noticed the problem with interrupted file transfers so it clearly wasn’t just an internet connectivity issue. It looked like something to do with the NIC, the cable or a network device.

The network trouble was accompanied by a series of System log event errors from MVSMP:

Port ‘SWITCHPORT-SM-F277C685-E5F8-490D-8CD1-913B854FABD2-0-1′ was prevented from using MAC address ’00-15-C5-7E-EB-39′ because it is pinned to port ‘SWITCHPORT-SM-F277C6′.

The only coverage I could find on similar topics suggested it might be a MAC conflict or a problem with the physical switch at the end of the wire.

The first thing I checked was the host’s external connection’s MAC address and whether it resided in the Hyper-V external network MAC pool (something I’d never inspected before). It turned out it wasn’t. Each laptop with this image had its own randomly-generated MAC pool, as the Hyper-V role was manually added post-deployment. In short, a MAC conflict on a Hyper-V MAC pool range seemed unlikely, especially since the host’s adapter wan’t even on that range.

Having exhausted the quick troubleshooting and investigative options, I set about fixing it, leaving the root cause for another day. These were the steps I took to resolve the issue in our environment, which assumes use of internet connection sharing (ICS) from a Hyper-V external network connection to a Hyper-V internal network connection:

Disable the ICS connection in any running guest VMs
Disable ICS in the host
Delete the external network in Hyper-V Manager
Uninstall the host’s physical NIC in Device Manager
Scan for new devices
Add a new External Network using the newly reinstalled device, called Hyper-V External Network in Virtual Network Manager
Rename the Local Area Connection in the host to Virtual Network Switch
Rename the new Local Area Connection for the host’s external network connection to Host External Network Connection
Share ICS from the Host External Network Connection to the Host ICS Network Connection
Re-enable the ICS adapter in the running guest VMs
Confirm that the System event log errors are resolved and that the connection appears to be stable

This has only happened once in a number of deployments of this build, so I suspect it’s a fluke. If it arises again it will be worth tracking MAC address allocation throughout this process, as the host external connection will change from the old conflicting MAC to the physical MAC when the adapter is reinstalled to a new MAC when the NIC is given to Hyper-V again. One assumes that if a new MAC is not generated the problem will persist, based on that error message.

Update (15/09/09): this has now happened twice more but I haven’t been able to pin down a cause still. In each case the fix has been straight-forward, as above and the problem has not returned on those machines.

Update (22/10/09): Further updates.