For clarity, I will quickly state the problem as I see it. SharePoint 2010 system requirements and practitioner mobility requirements are inherently at odds. What guidance exists for this unique problem space tends to regurgitate preferences/allegiances rather than comparing technologies and ratifying assumptions with real-world tests. At best, you get system performance indices for a single laptop model, but these results may vary when any hardware component is changed.

How can virtualisation improve system performance?

It doesn’t. People look to virtualisation to solve other problems. However, SharePoint 2010 performs differently in different virtualisation technologies, and the margins of these differences vary by hardware configuration. By all means, the advantages of virtualisation often make it a desirable choice, but these performance characteristics need to be accounted for, lest system performance losses negate the productivity improvements that virtualisation can introduce.

Why virtualise?

There are a number of advantages to virtual systems over physical systems. Many of these benefits can also be obtained with sufficiently mature systems management technologies and physical systems, but these benefits are often easier, quicker or less costly to implement through virtualisation. Some of the benefits include:

• Provisioning times for new SharePoint environments.
• Standardisation through cloned, network-isolated virtual machines.
• Account for volatility with snapshots.
• Standard builds per-project, to share with team members, reducing project initiation costs.
• Virtual appliances produced by Microsoft and third parties, such as the Information Worker Demo VM.
• Reduced hardware rebuilds by removing development tools and SharePoint from the host.

This list is by no means comprehensive. As I say, many of these benefits can be realised with scripting and/or management tools. This list is only intended to illustrate why it’s a powerful design option.

An overview of virtualisation and related technologies

Some example technologies by type:

• Type I Hypervisors
  • VMWare ESXi
  • Hyper-V
• Type II Hypervisors
  • Oracle VirtualBox
  • VMWare Workstation
• Infrastructure as a Service (IaaS)
  • Amazon EC2
  • Azure VM Role (forthcoming)
• Local Systems
  • Native Boot Windows 7 (virtual hard disk)
  • Citrix XenDesktop (VDI)

Note: Virtual PC was not included because it doesn’t support 64-bit guest operating systems. SharePoint 2010 only runs on 64-bit systems.

Some of the alleged benefits of these approaches:

• Type I Hypervisors
  • Better performance**
  • Good management options/tools
• Type II Hypervisors
  • Host Operating System
  • Easy to use
• Infrastructure as a Service (IaaS)
  • Pay as you go
  • Scalability
• Local Systems
  • Good performance
  • Simple to use

Some of the alleged drawbacks of these approaches:

• Type I Hypervisors
  • No Host Operating System***
  • Driver issues*
  • Complicated
• Type II Hypervisors
  • Historically poor performance**
  • Historically, less manageable (snapshots, import/export, etc)
• Infrastructure as a Service (IaaS)
  • Requires stable connectivity
  • Complicated
  • Pay-As-You-Go requires diligence
• Local Systems
  • Easy to damage
  • Slow to rebuild

*Hyper-V has driver issues on some newer laptops. These are most noticeable with graphics, although I have seen audio driver problems as well. Some of these driver issues may be fixed or alleviated in the SP1 Beta/RC for Windows Server 2008 R2.

**This performance bias is one of the things I will be examining in more detail in later posts.

***This is only “sort of” true for Hyper-V, which invokes a “parent partition”. This is a special type of virtual machine that fulfils a similar role to a host operating system, and is often referred to as such.

Why are “Local Systems” included?

I’ve lumped these in for two reasons. 1) They share some characteristics with the other virtualisation technologies, like running from virtual hard drives. 2) By virtue of being local systems, they fundamentally negate some of the benefits that are obtained through virtualisation. Cloning these machines is not an option if SharePoint is installed and configured. It will be necessary to invest in scripting environment provision in order to retain those productivity benefits. It happens that many people choose to take this scripting approach, but it’s worth pointing out that network isolation and cloning can achieve similar results through virtualisation, and this does not obtain with Local Systems.

What about shared, hosted development environments?

In this scenario I’m thinking of hosted development farms, where some or all members of a team use a single environment. Based on my subjective reading of the community, this option seems to be fading away. I think there are three reasons why.

1. Cost. Running development environments on proper infrastructure is expensive. Most components have been made redundant, the storage will be expensive if it performs well, the power/cooling costs are considerably more expensive than for laptops/desktops and you will need to pay people to manage the systems. Even when these costs are split across multiple developers, it’s still expensive unless resources are overcommitted, which negates productivity gains. It also tends to be more expensive to provision new environments and this process can often be an obstacle to business agility. In a nutshell, these are protections that are unnecessary for development environments. Redundancy and resilience at this level is overkill given the associated costs. The most important assets, such as code, standard images and project-specific builds can be protected separately.
2. Hive pollution. If these farms will support multiple projects, as they often do per the previous comments about provisioning, then these systems will inherently differ from the test/stage/UAT/production systems they should resemble. Core files in the hive can be altered from project-to-project, resulting in unexpected behaviour when moving code between these environments. This can seriously complicate troubleshooting and should be avoided.
3. Mobility. These farms aren’t terribly useful to people who are travelling or who are working on-site with restricted outbound connectivity.

All of this said, there are times when project-specific requirements may make shared farms a good option. It may be sensible to take another look for:

• Integration projects.
• Developing with large amounts of data.
• Projects with heavy infrastructure requirements, such as FAST.
  • Perhaps individual development environments can consume a shared FAST Service Application?

Generally speaking, I believe these resources should be provided only in these niche cases.

How is this different from IaaS?

The main differences are costs and capital. Cloud-based infrastructure services are fundamentally just virtualised hosting on an enormous scale. This scale lowers costs to a point where it may be affordable to deploy individual machines per-developer. Although in my analyses I found that IaaS would be more expensive than desktop workstations over three years, this still may be compelling when cash flow issues preclude significant one-time investment or credit flows are restricted. IaaS should also be kept in mind when specific projects require significant provisioning or investment for a short term, for instance testing in a large farm.

While providing a single cloud-based VM per-user solves the first two issues with shared development environments, mobility is still an issue. In many places, stable mobile broadband is flaky at best. Additionally, there are key architectural differences that need to be accounted for when working in the cloud, and on a Pay-As-You-Go basis. I address all of this in my series on SharePoint 2010 Infrastructure for Amazon EC2.

Which approach is best?

This is a high-level overview of the design constraints that limited my choices, before I plunged into a concrete performance review of the remaining technologies.

Local Systems
In my view, Local Systems are only a better choice if the supporting IT systems and processes are very mature and the performance benefits are clear and significant. For most development scenarios, that has yet to be proven. I’ve postponed this virtual to physical performance comparison for now, as the other benefits of virtualisation have ruled this approach out for us, but I hope to revisit it in the new year.

IaaS
IaaS has two key planning considerations. The first is fairly obvious. Outbound RDP Connectivity needs to be open whenever the systems are needed. I encourage people to consider this in some detail and pilot with many types of users before diving in. The second consideration is Pay-As-You-Go. While cloud providers often have an always-on option, it’s usually pretty pricey. The alternative is to find a mechanism to limit compute usage to when it is truly being used, without introducing usability problems. Management tools or scripting may be able to answer these problems, but no one should enter in to this process thinking it will be easy. This is not an easy option. For a more detailed consideration of these issues, refer to my series on EC2.

Type II Hypervisors
VMWare Workstation is the most mature desktop virtualisation product on the market, although in recent years VirtualBox has been gaining share. Choosing between these technologies for my tests was never going to be easy, but I reduced it to a few factors:

• I’ve never met a VirtualBox user that would complain about using VMWare but I can’t say that proposition is reversible. There are a lot of SharePoint practitioners with a strong preference for VMWare.
• VMWare Workstation has native interoperability with other VMWare assets. While VirtualBox supports the VMDK file format, it’s not quite the same thing.
• Both products are fairly inexpensive in the grand scheme of things.
• I had stability issues with VirtualBox circa version 3.14 that left a bad taste in my mouth.

Perhaps most importantly, I felt that the performance comparison of VMWare Workstation to Hyper-V would be the most valuable decision-making information.

Type I Hypervisors
Most Type I Hypervisors would not be suitable for desktop virtualisation because they don’t have a host operating system. While it would be possible to boot a guest OS and remote in to other Virtual Machines over internal networks, this is a complicated approach and the networking requirements would be enough to put off most developers. However, as mentioned above, Hyper-V is a notable pseudo-exception to this with its parent partition.

We’ve been using the Hyper-V role in Windows Server 2008 R2 for development for a little over a year now. While we have successfully capitalised on many of the productivity benefits of virtualisation through this approach, there are a few issues that have never been entirely satisfactory:

• Despite having the host OS, using Hyper-V is still complicated for non-Systems people – particularly the networking.
  • Work-around solutions for Wireless networking are fiddly.
  • Lack of self-contained NAT requires the use of Internet Connection Sharing in order to achieve network isolation, which some users struggle with.
• Lack of Sleep/Hibernate is painful for many users.
• Graphics performance is poor – particularly with large PowerPoint/Visio files, large images and video.
• Audio can also suffer during large file operations.
• Hyper-V is not ready for laptop power schemes.

Despite these niggles, we’ve continued to use Hyper-V while waiting for the forthcoming graphics/memory improvements in Windows Server 2008 R2 SP1. I would class these usability problems as significant inconveniences that sometimes manifest themselves in lost productivity – particularly with new users learning our approach.

New Problems in SharePoint 2010

Since we properly immersed ourselves in SharePoint 2010 development, negative reports about performance started to roll in. These proved hard to validate until a few months ago when my colleagues showed me first page load times after an IISRESET in excess of one minute. This was concrete and repeatable. The problem was more severe on some systems than others, but it was clearly a problem.

The performance tests I’ve been conducting have been an effort to pick apart these results in Hyper-V. Was this new in SharePoint 2010 or did it amplify something that was minor before? Do we get the same problems on different virtualisation technologies, in the cloud or is this a symptom of virtualisation itself? In my next post I’ll discuss the environments, the tests and the testing process.

Exchange Server Reconfiguration

Tidying up the Exchange server is a much more straight-forward process. In fact, all of the changes that I made are network orientated per the network changes from the first post, so if you are not adding a second NIC or a second fixed IP address on the original internal NIC, these steps aren’t necessary.

• Import the virtual machine. Plug the Hyper-V Internal Network in to the first NIC and add a second NIC with the Hyper-V ICS Network plugged in to it.
• The initial IP address is 192.168.150.2.
• Rename the Local Area Connection NIC to Hyper-V Internal Connection (or your preference).
• Rename the Local Area Connection 2 (or maybe 3) NIC to Hyper-V ICS Connection (or your preference).
• In IPv4 properties on the Hyper-V ICS Connection, change the Advanced TCP/IP Settings to not “Register this connection’s addresses in DNS”, as it is dynamic.
• I disabled IPv6 on both NICs as it is already disable on our host’s network connections.
• Add a second IPv4 address to the Hyper-V Internal Connection:  192.168.200.151/255.255.255.0.
  • Added the same address on the DNS tab for this NIC.

• Check DNS to make sure the new address was added.
• Added a HOSTS file entry for demo2010b pointing at 192.168.200.151.
• Tested logging on via Remote Desktop from the Host machine.

Shutdown/snapshot and optionally export the VM if time/resources permit.

Reviewing SharePoint Server event logs

This section details my review of the event logs in the shipped state. I did not take any action except for the last two items regarding DCOM fixes and the SharePoint Health Logs. I believe that most of these errors are probably an effect of running so many things in one environment, but I’d welcome comments if you have any insight to share.

• SetSPN for WSMAN warnings: The WinRM service failed to create the following SPNs: WSMAN/demo2010a.contoso.com; WSMAN/demo2010a.
  Additional Data
  The error received was 8344: %%8344.
  User Action
  The SPNs can be created by an administrator using setspn.exe utility.
  Presumably these SPNs can be created manually or the rights to create the SPNs can be assigned to the WinRM service account if needed, but I am not making any changes here until I see that it is necessary to do so. There are some Kerberos Audit Failures in the security logs but since the SharePoint environment is self-contained and there are no secondary hops, I don’t think this is worthwhile.
• VSS Error 8320: Volume Shadow Copy Service error: Failed resolving account Administrator with status 1376. Check connection to domain controller and VssAccessControl registry key.
  Operation:
  Initializing Writer
  Context:
  Writer Class Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
  Writer Name: WMI Writer
  Error-specific details:
  Error: NetLocalGroupGetMemebers(Administrator), 0×80070560, The specified local group does not exist.
  There are a number of articles that discuss fixes for this Warning but since this is an unusual configuration (SharePoint/SQL on a DC) and the warning is about the “Administrator” account, I am hesitant to make this change, for fear of introducing instability. Further reading at Experts Exchange, Technet and AnandTech.
• There are two errors regarding the ULS config file, 7105 and 7056. I’m honestly unsure what to make of these errors since they refer to C:\Program Files\Common Files\Microsoft Shared\ULS\14\uls.config.xml and I’m not certain how that is used in generating the ULS log files for SharePoint which are appearing normally at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\LOGS. Not taking any action for now.
• Microsoft.ResourceManagement.ServiceHealthSource Error 22:
  The Forefront Identity Manager Service cannot connect to the SQL Database Server.
  The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.
  Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, verify the connection information has been configured properly. This configuration is stored in the Windows Registry.
  This error seems to occurs before the FIM Sync service starts, which may or may not be a related issue. The User Profile Service Application is available though, so I believe this has something to do with this connection attempt occurring before the Service Application is able to connect with SQL.
• CAPI2 Error:
  Failed extract of third-party root list from auto update cab at: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab> with error: A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.
  This issue seems to have emerged in July of this year on Windows 7 and Windows Server 2008 R2 machines. Unfortunately there’s no clear resolution at this time, but also no clear negative impact on this system, so I’m not taking any action for now.
• DCOM 10016 error. A fix for this has been documented well by Matt Groves.
• SharePoint Health Reports:
  • Almost all of the warnings and errors in the Health Report are either one-time issues that no longer exist or are symptoms of the single-server install on a DC.
  • An exception to this is the “Validate the My Site Host and individual My Sites are on a dedicated Web application and separate URL domain” message, which is accurate. The environment has not been configured with a dedicated MySite application.
  • The one other issue was “The Unattended Service Account Application ID is not specified or has an invalid value”. This has not been set up in the VM and will need to be configured in the Secure Store Service if it will be used.
  • I have chosen to disable all of these rules, as they are undesirable in a demo environment. I also deleted the existing alerts.

Pending

As I mentioned above, I hope to revisit this VM and create new snapshots that will reduce load by disabling bulky services. This snapshot branch might resemble something like this:

• Current state, as above
  • Most SharePoint Services turned off in Services on Server
  • BI Indexing Connector Service , FAST, OCS and Project Server removed. All SharePoint Services turned on
  • As above, but with FAST
  • As above, but with OCS
  • As above, but with Project Server
  • BI Indexing Connector Service , FAST, OCS and Project Server removed. All SharePoint Services turned off
  • As above, but with Search/FAST
  • As above, but with OCS
  • As above, but with Project Server

I really haven’t planned this yet or discussed these options with users of this environment, so it could wind up looking completely different. Also, this many snapshots would probably chew up too much disk space and get confusing for the users. But watch this space for updates as I hope to revisit the topic again. And please feel free to suggest other optimisations or additions that work well for you.

Now the public beta trial is expiring and people are moving to the RTM build. It appears to be much improved, in that more of the product works in this version and a few niggles have been fixed now. However, it’s widely acknowledged that the resource requirements for this virtual machine are gargantuan due to the breadth of what it offers.

What you get

The first of these VMs doesn’t just have SharePoint Server 2010 (Enterprise). It has:

• Active Directory Domain Services
• DNS
• SQL 2008 R2
• Office Professional Plus 2010 (including Visio and Project)
• SharePoint Designer 2010
• Visual Studio 2010
• Office Web Applications
• FAST Search for SharePoint 2010
• Project Server 2010
• Office Communication Server 2007 R2

There is a second VM with Exchange Server 2010 which you can use as needed. Note: when downloading the RAR files which self-extract to the VM export files, I highly recommend using the Akamai download links. They will save you time and frustration.

Why I’m writing about it

The first of these VMs does a lot, and will chew up more virtual resources than most physical demo machines have to provide. In this post I documented the tweaks I made to our copy of the Virtual Machines and I’ve reviewed the event logs to identify any other issues that are present in the shipped state. I also spent some time fixing the start-up of some services due to observed delays in network connectivity.

The optimisations achieved here are not immense but they establish a firmer baseline from which to carry out further improvements. Unfortunately I have not had the time to explore the best approach to taking these core assets and removing the bulky bits in order to provide snapshots that unleash bits of the functionality as needed. With time I hope to revisit this and potentially explore other uses for this environment.

SharePoint Server Reconfiguration

These are the steps I’ve taken to reconfigure the SharePoint Virtual Machine:

• Imported the virtual machine. Plugged the Hyper-V Internal Network in to the first NIC and added a second NIC with the Hyper-V ICS Network plugged in to it. To understand more about the NIC naming conventions I’m using here and how we use these Hyper-V networks, please refer to my series on SharePoint development environments.
  • If you’re curious, we dual-boot our Sales laptops in to this development environment so our sales people can use Hyper-V as needed.
• The default SharePoint VM RAM is set to 5120MB (5GB) RAM. This could be increased up to ~6GB on a host with 8GB RAM, assuming the load on the root partition is minimal during demonstration and the Exchange VM is not being used concurrently.
• If using a UK keyboard through the Hyper-V Virtual Machine Connection, the password will need to be entered as pass“word1
• Note: at login a warm-up script runs, which I will discuss more in a moment.
• I changed the time zone to London.
• I changed UAC to “Notify me only when programs try to make change to my computer (do not dim my desktop)”. I find that with Known Hyper-V graphics performance issues this setting achieves the right balance between usability and security.

Service start-up

• I reviewed the event logs to identify which Manual Services would start up at a delay. Approximately five minutes after the machine started the Virtual Disk Service started up. That was followed by the following services:

  Service	Startup Type
  Application Experience	Manual
  BI Indexing Connector Service	Automatic (Delayed Start)
  Certificate Propagation	Manual
  Diagnostic Policy Service	Automatic (Delayed Start)
  Diagnostic System Host	Manual
  Distributed Transaction Coordinator	Automatic (Delayed Start)
  FAST Search for SharePoint	Automatic
  FAST Search for SharePoint Browser Engine	Manual
  FAST Search for SharePoint QRProxy	Manual
  FAST Search for SharePoint Sam Admin	Manual
  FAST Search for SharePoint Sam Worker	Manual
  Function Discovery Provider Host	Manual
  IPsec Policy Agent	Manual
  Microsoft .Net Framework NGEN v4.0.30319_X86	Automatic (Delayed Start)
  Microsoft .Net Framework NGEN v4.0.30319_X64	Automatic (Delayed Start)
  Network Connections	Manual
  Network List Service	Manual
  Office 64 Source Engine	Manual
  Office Software Protection Platform	Manual
  Portable Device Enumerator Service	Manual
  Remote Access Connection Manager	Manual
  Remote Desktop Services	Manual
  Remote Desktop Services Configuration	Manual
  Remote Desktop Services UserMode Port Redirector	Manual
  Secure Socket Tunneling Protocol Service	Manual
  SPP Notification Service	Manual
  SQL Full-Text Daemon Launcher	Manual
  Telephony	Manual
  Windows Defender	Automatic (Delayed Start)
  Windows Module Installer	Manual
  Windows Remote Management	Automatic (Delayed Start)
  Windows Search	Automatic (Delayed Start)
  Windows Update	Automatic (Delayed Start)
  WinHTTP Web Proxy Auto-Discovery Service	Manual

• In my testing, switching the Network Connections and WinHTTP Web Proxy Auto-Discovery Service services to Automatic would reduce time to CTRL+ALT+DEL from five minutes to three minutes. It’s worth verifying these results in your environment.

Service Hardening

I disable the following services, per my normal virtual machine hardening processes. I don’t believe that any of them are necessary in this environment:

• Certificate Propagation
• Desktop Windows Manager Session Manager
• Distributed Link Tracking Client
• Encrypted File System
• Function Discovery Provider Host
• Function Discovery Resource Publication
• IP Helper
• Microsoft iSCSI Initiator Service
• Multimedia Class Scheduler
• Problem Reports and Solutions Control Panel Support
• Remote Procedure Call (RPC) Locator
• Smart Card
• Smart Card Removal Policy
• Special Administration Console Helper
• Tablet PC Input Service
• Windows Audio
• Windows Audio Endpoint Builder
• Windows Error Reporting Service
• Windows Search
• Wired AutoConfig

I also disabled these services. I have no idea why they were ever enabled.

• Microsoft Fibre Channel Platform Registration Service
• Microsoft FTP Service

Following a reboot to assess boot times after hardening, I was prompted with a firewall exception warning for SharePoint Workspace. I allowed this exception.

MSCONFIG

I mentioned earlier that a warm-up script launched at login. I reviewed the contents of the script and identified that it enumerated all sites in the farm, so I browsed to each site that was getting warmed up to have a look at that content. Most of these sites pull up the “select a template” page, indicating that there is not content in the web application, so I have chosen to disable the script for now and will likely revisit the use of the IIS.NET application warm-up module when it matures. In the mean time, I recommend that recipients of this build take a snapshot of the VM while the http://intranet and Central Admin web applications are fully warmed up for their needs so they can roll back to that point as needed.

While in MCSONFIG, I turned off:

• Watson Subscriber for SENS Network Notifications
• EcmCopyLinks
• Warm Up
• Optional:
  • Microsoft Office 2010 (these are sync services – probably best to leave them running)
  • Microsoft Office Communicator 2007 R2 (this is preferential)

Housekeeping and simple benchmarks

At this point I wanted to take some basic start-up timings so I shut down the VM, but before starting back up I changed the Hyper-V VM’s BIOS settings to boot from IDE before CD. After that I took some pretty unscientific measurements of boot times and resource consumption on my Dell XPS M1330:

• CTRL+ALT+DEL: 2:56.
• Desktop: 3:47.
• Baseline RAM consumption down from >4.5GB to ~3.65GB before hitting the intranet and Central Administration web apps (remember, they aren’t warmed up anymore).
• RAM consumption at just under 4GB after hitting Central Administration.
• RAM consumption at ~4.25GB after hitting http://intranet.

After taking these measurements I shut down the VM and took a snapshot.

Network changes

These changes are optional, depending on how you will use the VM in your environment. I opted to add a second IP address on our existing internal network range, as follows:

• The initial IP address is 192.168.150.1/255.255.255.0.
• Rename the Local Area Connection NIC to Hyper-V Internal Connection (or your preference).
• Rename the Local Area Connection 2 (or maybe 3) NIC to Hyper-V ICS Connection (or your preference).
• In IPv4 properties on the Hyper-V ICS Connection, change the Advanced TCP/IP Settings to not “Register this connection’s addresses in DNS”, as it is dynamic.
• I disabled IPv6 on both NICs as it is already disable on our host’s network connections.
• Add a second IPv4 address to the Hyper-V Internal Connection: 192.168.200.150/255.255.255.0.
  • I also added 192.168.200.150 on the DNS tab for this NIC.
• Check DNS to make sure the new address is added.
• Add a HOSTS file entry in the root partition for demo2010a pointing at 192.168.200.150.
  • If desirable, add another entry pointing for intranet.contoso.com pointing at 192.168.200.150. This can be used for browsing from the root partition’s browser.
• Tested logging on via Remote Desktop and browsing to Central Admin from the Host machine.

Client tools and other changes

Some final changes are optional, but I think generally desirable.

• Install Firefox, Opera, Safari, Chrome.
• Install PDF Exchange Viewer or your choice of PDF viewer.
• Update: As mentioned by Leon Zandman in the comments here, the free SysInternals ZoomIt tool is very useful for presentations. You may also want to consider adding it.
• Add the environment variable path to the 14 hive’s BIN.

And that’s it for the SharePoint VM. Shutdown and take a new snapshot, optionally deleting the first one. I would suggest exporting the VM if time/resources permit, noting that export operations can be time-consuming and disk intensive.

In part two I’ll discuss the Exchange VM, event logs and potential future improvements.

After that I tried to delete other snapshots but I kept getting errors and the VM entered a Saved-Critical state. This will happen when Hyper-V Manager cannot access a file system location or cannot find a file, for instance when a removable hard drive is pulled out.

Approximately 30 seconds later, Hyper-V thought that it regained access to the location:

However, I couldn’t get any snapshots to delete and the virtual machine wouldn’t start. After a few minutes of panicked clicking I decided to restart the Hyper-V services. When they came back up my VM disappeared. The Virtual Machine configuration file was corrupted.

The next event suggests that a snapshot file was also corrupted.

These 16310 and 16330 errors repeated for a while. Panic continued. Eventually I rebooted. On reboot the VM was still missing and the 16310/16330 errors persisted.

On a hunch I decided to see what the AVHD files (the differencing disks that correlate with snapshot states) looked like.

This looked very much like what I would have expected if nothing had gone wrong (and if none of the snapshot deletions completed). Sticking with this line of inquiry (and what the 16310 error suggests), I created a new virtual machine and pointed it at the most recent AVHD file (selected above). All of my snapshots were missing but the virtual machine created successfully. I started the virtual machine and it was clearly in the same state it was in before I took the most recent snapshot, with a few caveats. In my panic I forgot to re-create my second NIC, so the VM started with only one (the one that I specified when I created the new VM). I also forgot to give it a second CPU. So I shut down the VM, made these changes, restarted, reconfigured the second NIC and tested that everything worked to my expectations. Recovery complete, so I shut down both VMs again.

At this point I’d recovered the VM but I still had a bunch of unnecessary data in my branch of differencing disks. In order to clean this up, I took a new snapshot of both VMs and exported the latest snapshot of each of them. This merged all the differences across the AVHD files in to a new, self-contained VHD file. After the exports finished I deleted the old VMs, waited for the Destroy operations to complete, cleaned up lingering files on the file system and imported the new exports. I took a new snapshot, as this is my new stable starting point and everything was (relatively speaking) back to normal. Phew!

With hindsight, I would have handled the recovery as follows:

• Create new VM, pointing at latest differencing disk (or whichever snapshot state you wanted to preserve).
• Reconfigure processors.
• Reconfigure network adapters in Hyper-V Manager.
• Start the virtual machine.
• Reconfigure NICs in the guest.
• Reboot.
• Test everything is working as expected in the VM.
• Shut Down.
• Snapshot.
• Export.
• Delete old VM from Hyper-V Manager.
• Wait for the Destroy operation to complete.
• Delete any lingering files from the file system.
• Import the exported virtual machines.

As I hinted at above, having gone through this process, it occurred to me that you could probably point at whichever AVHD file you wanted to, if you didn’t want to use the latest snapshot, assuming none of the AVHD files were corrupted. In this case it was just the virtual machine XML file and possibly the snapshot file that were corrupted, rather than the VHD file and differencing disks (AVHD files) themselves. The problem would be identifying which AVHD file corresponds to the snapshot that you want to keep, but in principal I think this would work.

I should note that this is probably unsupported, but you’re not really losing anything because otherwise you would have only been able to recover the first VHD file. This technique wouldn’t be much use if you didn’t know which snapshot you were after or if you wanted to recover the entire snapshot tree, but this fix gives you some recovery where the virtual machine file and the snapshot tree are corrupted but the disk data is not.

Cloning isolated VMs vs. scripted installation

One of the challenges we’ve always faced with SharePoint development has been the tension between cloning actually identical environments versus automating the deployment across distinct environments (or worse, repeating the installation manually). In the first case we save time by eliminating reconfiguration and this ensures a consistent experience for each user. This is particularly beneficial for software development. These benefits can also be obtained by scripting installation/configuration/deployment but there’s a considerable overhead associated with developing and testing those scripts. As SharePoint 2010 is still quite new and we’ve been working on projects for some time now, we didn’t have the luxury of waiting for those refinements and we needed to take advantage of these efficiencies as we had done with SharePoint 2007 projects.

This cloning approach has one big drawback: it requires network isolation in order to prevent network, machine or SID duplication issues. We can make up for that partially by using Network Address Translation (NAT) so that we can get outbound connectivity from the VMs (we do this using Internet Connection Sharing to an internal network), but this doesn’t provide any inbound connectivity for Remote Desktop connections to the VMs or the SharePoint applications in them. Remote Desktop access may be desirable because of the limitations of the Hyper-V Virtual Machine Connection or because the users do not have permissions to Hyper-V itself. Alternately, it may be desirable to expose the virtual machine’s browser or another application directly, as I’ll describe later in this article.

Before I detail the approach to publishing a network-isolated remote desktop connection, I should note that it’s also possible to lock down the environment using Windows Firewall and Hyper-V networking so that inbound traffic is isolated enough that most duplication issues are accounted for – but this requires considerable planning, additional reconfiguration and it’s a large enough topic that it requires another post (I will try to revisit this soon). In short, I think each approach addresses different needs. For now, I’ll talk about RemoteApp.

RemoteApp to a Remote Desktop

It’s been some time since I looked at the new features in Windows Server 2008 R2. I spent most of last Summer getting to grips with them but have been living in a world of SharePoint 2010 since, so I never got a chance to revisit the interesting developments with Remote Desktop Services. Among them is RemoteApp, which allows us to publish applications (rather than a terminal session) on a server through Remote Desktop Web Access. That application could be Word, PhotoShop, SQL Server Management Studio, ULS Viewer or whatever might be useful.

In the last couple of weeks I’ve addressed a couple of requirements by publishing a saved Remote Desktop connection file through Remote Desktop Web Access. To give a brief overview of the solution, I’ve logged on to the Hyper-V host server and saved a Remote Desktop connection file pointed at the target Virtual Machine. This relies on the Hyper-V host’s internal network connection to the virtual machine and our LAN connection to the Hyper-V host (the VM does not have an external network connection). Then I published that saved remote desktop connection file using RemoteApp. Users launch the connection through the Remote Desktop Web Application. This is a very basic example of how RemoteApp can help us by bridging a network barrier that we have to maintain.

RemoteApp to a Remote Application

If we get slightly more complex we can also get to the real beauty of this approach. RDP connection files can now be customised so that you can expose an application inside of the target machine. This is like Windows 7′s XP Mode, except I’m able to take advantage of the remote machine’s applications through Remote Desktop Services without having to log on to either the Hyper-V server’s terminal or the virtual machine. Put another way, I can log on to my Hyper-V server’s Remote Desktop Web Application portal and launch an RDP file. That RDP file bypasses both terminals and provides me with the application that I’m really after – say, ULS Viewer, or a browser that can render the network-isolated SharePoint data. Here’s a more detailed scenario:

• A Hyper-V server on the network has a number of virtual machines that fully replicate our domain infrastructure.
  • Because this is an exact copy of our live environment we need to keep the test environment completely network-isolated.
• We’re testing Word 2010 on multiple operating systems before rolling it out across the network.
• The tester needs to be able to access the Word Application but should not have access to the rest of the infrastructure.
• We have XP, Vista and Windows 7 VMs on the Hyper-V server – all running Word 2010.
• We log on to the Hyper-V server and create saved RDP files to each of those VMs.
  • We modify those RDP files so that they expose only the Word application. This is a few lines of added or modified XML in the connection file and a change to the TsAppAllowList registry key on each virtual machine. More detail on that approach here.
• We add each of these saved RDP files as Remote Applications in the RemoteApp Manager Remote Desktop Services node on the Hyper-V server. To be clear, this is new Remote Desktop Services functionality. It only happens to be a Hyper-V server because we’re exposing Hyper-V guests.
• The tester logs on to the Remote Desktop Web Access site and launches the RDP files as needed. Each file will expose an instance of Word 2010 which is running in the remote (and isolated) virtual machine’s context.

This experience is improved if you install the Hyper-V server’s certificate for the Remote Desktop Web Application site. Also note, you will be prompted twice to authenticate – once to the Hyper-V server and once to the virtual machine. That’s not a great user experience, but it’s also not the worst. To be honest, I haven’t had the time to see if it can be improved with Kerberos or another approach. I’d be interested to hear more if others have deployed this approach for more critical loads or more demanding users.

Other uses

There are loads of potential uses for desktop virtualisation like this. One of the most obvious ways that it could be used would be to allow a team member to access an isolated development environment (or one of its applications) over the LAN. This wasn’t possible with my earlier design, and frankly, you don’t want to do a lot of this, but sometimes it can be really useful.

I realise this is conceptually difficult, so feel free to ask questions if this hasn’t been my clearest post. I’ve skipped some steps that are required to deploy the Remote Desktop Services. That’s another topic that is beyond the scope of this post. Again, I’m happy to answer questions about that, although I’d recommend hitting TechNet for a few hours before diving in. There’s loads of new functionality here and you may find another option suits your needs better. To the initiate, the Microsoft App-V, MED-V, MDOP story might look a bit “nice to have”, or possibly too complex, but this really isn’t complicated to implement and engaging with this new technology has opened my mind to other design options that I had previously shut out. Definitely worth a look.

A couple of weeks ago I was working simultaneously on my Windows Server 2008 R2 laptop with Hyper-V (the same laptop build that’s been previously mentioned) and a Windows 7 x64 build that I was using for testing, when I noticed severe but intermittent network problems on both machines. After a fair amount of head scratching, I noticed that the two laptops had duplicated MAC addresses. Blatantly that shouldn’t happen, as the whole point of a MAC address is to provide uniqueness. The most perplexing issue was that the addresses conflicted across two different operating systems. However, it happened. Both wired adapters on the two machines had the MAC address 00-21-9B-DC-8E-0B. I uninstalled the wired adapter on the Windows 7 machine and scanned for new hardware. When the device reinstalled the problem went away.

Since then I’ve been poking around a bit and I’ve found this old TechNet article from December 2007 in which Wes Miller says:

In addition to changing the SID and the machine name, you also need to change certain values that may be specific to the virtual computing technology you’re using. In particular, you need to change the MAC address (the unique ID for networking devices). Plus, many virtual applications also have their own unique identifier. Most store these in their own machine configuration files, so you’ll want to know how to manipulate those entries (and maintain their validity). Note that many virtualization products that support Pre-Boot Execution Environment (PXE) key the SMBIOS UUID based on their own unique ID—emphasizing the need to change this (or let the virtualization software change it for you, if supported) if you’re joining it to a domain; otherwise, managing WDS or RIS-client systems can become impossible (if GUIDs conflict). Most of the virtualization solutions I’ve worked with can have severe networking problems in the case of duplicate MAC addresses; so if you are not just moving a virtual machine, it’s very important that you change the MAC address if the virtualization software does not do it for you.

That’s by no means conclusive, but I’m not turning up much else. So if you’re encountering network problems with WDS images that began their life as a virtual machine, you may want to consider automating the un/re-installation of the NICs post-deployment or addressing that with WAIK somehow. Otherwise just make sure to do it manually before giving the machine to the recipient, or people will wind up with a rather frustrating network issue.

If we get around to doing any more testing of this or gather more evidence from internal deployments, I’ll post the findings here. In the mean time I thought I’d update the ongoing saga here.

I’d actually geared myself up for this approach with the release of Windows Server 2008 R2 RTM, since Windows Deployment Services supports deployment of VHDs now, but I deflated myself a bit when I realised this was only a means of deploying for native boot from VHD rather than deploying a VHD to hardware as though it was a captured WIM. When I figured this out I went back to capturing physical images, and blindly overlooked this option. Nice one!

A few months ago I reminded myself of a major gotcha when planning a virtual infrastructure. Assume that you run more than one domain in more than one forest and that trusts are in place to authenticate users across those forests. This could be a development/test/staging environment, or as will no doubt be more common in the coming years, it could be a virtualised infrastructure.

Assume that Kerberos needs to operate across these domains for any number of purposes, from application authentication to Active Directory migration. If you want Kerberos to work, you’re going to need to synchronise time across these domains, as the clock synchronisation is used to time stamp tickets in order to prevent replay attacks. Time synchronisation is of course built in to the Windows domain infrastructure, and should support this nicely. Configuring the Windows Time service on a PDC emulator is a bit fiddly, but should be achievable for anyone who runs a multiple domain infrastructure. But what if it goes wrong?

I spent a few hours bashing my head against this utterly confounding problem until the obvious whacked me in the face. The problem:

• I would run w32tm /resync /rediscover and all would synchronise successfully, then about five seconds later the clock would revert to its former time, about 2 minutes and 45 seconds ahead of the recently received absolute value
• There were no error logs, just successful resynchronisation messages and then nothing to indicate why the clock reverted
• I set the registry keys to enable debug logging and this revealed nothing more than that the time of the events was shifting as you would expect

Then it hit me. The PDC emulator was a virtual machine and the host virtual server was in another domain/forest. That host server was failing time synchronisation with its DC, so I manually reset the host clock and voila! The PDC emulator was synchronised within seconds. Obviously there’s another task to find out why the host was failing synchronisation with its time server, but that’s totally beside the point.

The lesson: Obey the best practice guidance and turn off Host Time Synchronisation for virtual domain controllers. In Virtual Server 2005 R2 the setting is in the Virtual Machine Addition Properties.

And a follow-up note: once Host Time Synchronisation is disabled you will need to find a new source of reliable clock for the the VMs, as they don’t have a CMOS clock to rely on. You could use something as simple and free as Atomic Clock Sync. If you fail to do this the VMs will lose time when they are shut down.