• SharePoint Development Productivity and Virtualisation Technologies
• SharePoint 2010 Development Environment Performance Tests

I’ve said my preamble in those posts, so I’ll cut to the chase here.

High-Level Summary of Findings

• Disk performance and bus speed did not prove to be significant factors in these results (except for virtual machine start-up times). Obviously there are fundamental differences about SSD (yet untested) that may skew this picture, but I will be surprised to see big differences. If we’ve got these tests right, and they are actually representative of the tasks that slow down development, then we would expect to see wider variance across bus or disk speeds. We don’t.
  • This assumes the disk is relatively uncontended. Virtual machine performance degrades in every type of test while large file operations are running concurrently on the same disk. This could be copying an ISO, importing or exporting a virtual machine or any other sustained large file operations.
    • At a minimum, this is certainly an argument for running VMs on their own spindle, whether it’s over USB, eSATA or SATA. This may be an area where SSD shines.
  • These disk performance figures can be found towards the bottom of this post. Desktop performance was nearly identical running on USB2 at 5400 RPM versus a RAID0 stripe or a RAID1 array on 7200 RPM disks. Laptop performance was also nearly identical over USB2 5400 RPM versus eSATA 7200 RPM.
• Hyper-V performance has been poor on all laptops with i-Series CPUs. This is more pronounced in some areas than others. Our three-year-old model with a Core 2 Duo actually outperforms the new i7 in some cases. When these results are added to known driver issues with Hyper-V on many newer laptop GPUs, we’re looking at a configuration that’s unfit for SharePoint 2010 development.
• VMWare Workstation outperforms Hyper-V on laptops by significant margins in most areas. The exceptions to this are start-up time and performance during the first 10-30 minutes of use (I believe VMWare is ballooning during this time). After that, VMWare Workstation is faster than Hyper-V in every type of test.
  • As a long-time advocate of Hyper-V despite usability deficiencies, I was probably more surprised by the significance of these differences than anyone. I wrongly assumed that Type-I hypervisors would outperform Type-II in nearly every way. While that may hold true on server class hardware, it doesn’t hold true here. I’m a convert.
• While less pronounced, these same findings hold true on the desktop.
  • Desktop performance is very quick on VMWare Workstation, considerably out-performing even Amazon EC2.
  • We can realise significant productivity gains by moving all users who are primarily office-based to a desktop + VMWare Workstation configuration from laptop + Hyper-V, at a fairly small cost (probably half the cost of EC2 over three years – see my recent posts on EC2 for more information).
  • Desktop performance on Hyper-V, while notably slower than VMWare Workstation, is generally faster than VMWare Workstation on the i7 laptop.
• Laptop performance is significantly improved on our current model with VMWare Workstation. These improvements are also realised on the newer model laptop, but the performance delta between the two physical systems is not so significant that it’s compelling to move to a low speed i7 from a reasonable speed Core 2 Duo.
  • The total times for the “End-to-end site creation to debugging tests” were two and a half minutes faster with VMWare Workstation compared to Hyper-V on the Dell XPS M1330. Moving from Hyper-V to VMWare Workstation for laptop users is now an obvious choice.
  • The benefit of spending on i7 processors is in doubt. We are seeing very minor performance penalties when adding more than two CPUs in both VMWare Workstation and Hyper-V for most tests. There were also very minor improvements for some tasks, but on the whole there does not appear to be a measurable benefit. This might vary if the host OS is doing a great deal with the CPU, but that is liable to cause other contention issues than just in the CPU (on a laptop).
  • The only tasks that appeared to use all 8 cores in a SharePoint VM were:
    • Retract/Deploy of a solution (but only very briefly)
    • Create web app, or Create site collection (but at low percentages)
    • Rebuild with Code Analysis (but not fully)
  • We will be running future tests on i5 processors at higher clock speeds to see how these models perform relative to the 1.6 GHz i7.
• The User Profile Service Connection doubles first page load times after an IISRESET in all test cases. I consider this a full validation of these preliminary findings.

Snapshot of key data

The Data

How to read the data:

• Hardware: the physical laptop or desktop model (or Amazon’s EC2)
• Virtualisation: “Hyper-V” is short-hand for the Hyper-V role in Windows Server 2008 R2. “VMWare 7.1.2″ is short-hand for VMWare Workstation.
• #CPU: the number of physical CPU presented to the guest operating systems. Multiple logical cores were only tested in the 4×2 results below.
• Disk: the physical disk configuration where the virtual hard drives are running.
• RAM: the amount of RAM running inside the SharePoint Server 2010 VM. The Amazon EC2 instances were “large instances” but the domain controller was running locally.
• Test: The tests have been described in more detail in my last post.
• Result 1, 2, 3: Each test was carried out three times. The far-right column, Average Result, is an average of the three.
• The Two “Average Load…” rows are an average per-result of the three rows above them. These are tests built on SharePoint 2010 default site templates, which anyone should be able to replicate.
• The “Total create to debug time” row is a sum of the five rows above it.
• All results are in seconds. In cell G21 below, 524 seconds = 9 minutes and 2 seconds.
• For more information on the tests and the testing methodology, see my last post.

Hyper-V versus VMWare tests, all other things being equal

Dell XPS M1330, running Hyper-V

Dell Studio XPS 1645 laptop, running Hyper-V

ASUS V7-P7H55E desktop, running Hyper-V
Note: these Hyper-V tests were accidentally carried out while the VM was running on a RAID 0 stripe rather than on the System disk, so this is not apples and apples, but later disk tests on VMWare Workstation indicated that this shouldn’t make much of a difference, so I’ve left these results in, with this comment.

Dell XPS M1330, running VMWare Workstation

Dell Studio XPS 1645 laptop, running VMWare Workstation

ASUS V7-P7H55E desktop, running VMWare Workstation

VMWare Workstation i7 tests with 4 or 8 cores

Dell Studio XPS 1645 laptop, running VMWare Workstation with 4 CPU

ASUS V7-P7H55E desktop, running VMWare Workstation with 4 CPU

Dell Studio XPS 1645 laptop, running VMWare Workstation with 4 CPU, 2 Cores Each

ASUS V7-P7H55E desktop, running VMWare Workstation with 4 CPU, 2 Cores Each

Amazon EC2 Results

Notes:

• Times were much slower one day than others. This hasn’t been measured over time, but it’s worth keeping in mind. Other EC2 users reported similar problems on the same day.
• Also note: a couple of rows of test data (245 and 248) have been accidentally deleted, but the results were not unexpected in any way.
• Row 263 has no data because measuring time to desktop with EC2 would be too imprecise. It would normally be available within five minutes from start, for reference.

Disk Tests on VMWare Workstation with two cores

The format of these tests change slightly, as I am grouping all disk permutations for the Dell Studio XPS 1645 together, then moving on to the ASUS V7-P7H55E desktop. I grouped them this way because the tests were fundamentally different for laptops and desktops. I did not get the time to repeat the laptop tests on the Dell XPS M1330.

Dell Studio XPS 1645 laptop with VM running on 5400 RPM USB2

Dell Studio XPS 1645 laptop with VM running on 7200 RPM eSATA

ASUS V7-P7H55E desktop with VM running on 5400 RPM USB2

ASUS V7-P7H55E desktop with VM running on a 2nd set of RAID 0 spindles

ASUS V7-P7H55E desktop with VM running on a 2nd set of RAID 1 spindles

…and with that, I’ll let you draw your own conclusions. Should anyone want to contribute supplementary test data in the comments here, or carry out further tests (perhaps with SSD), I would love to see the results. As I mentioned in the last post, there’s still more testing to do.

Update 08 June 2011:SharePoint 2010 Development Environment Performance: SSD, i5 vs. i7, WEI and Sandy Bridge

The Tests

The 21 tests that we settled on were the result of discussions with a number of the core developers, consultants and architects at Content and Code, plus a few tests that I threw in to confirm/disconfirm some of my suppositions, such as the impact of the User Profile Service Connection on first page load time. All 21 tests were run three times for each permutation of hardware candidate and virtualisation technology. We also tested on Amazon EC2. I will discuss the testing process in more detail in a moment.

These tests have been selected for a few reasons:

• They are tests that anyone can run, including Visual-Studio-allergic types like myself.
• They re-enact real-world productivity loss. All tests needed to be significant on our current system or they were thrown out.
• They needed to account for tasks that impact non-developers as well as people that have their head down in code 40 hours/week.
• They needed to be examples of tests that would stress systems in different ways.

First page load tests
These tests were designed to examine what, if any impact different sets of features, functionality and structure might have on first page load times after the application pool is recycled or IIS is reset (while gathering a large set of data to make comparisons across systems). I also wanted to fully validate my preliminary findings about the User Profile Service Connection.

I ran these tests against NTLM-authenticated web applications with the following root site collections:

• Central Administration
• Blank Site
• MySite
• Blank Site, with no User Profile Service Connection
• The Content and Code website solution (structure, without content)
• A custom intranet solution (structure, without content)

All of these first page load tests were repeated for application pool recycles and IIS resets.

End-to-end site creation to debugging tests
I hope these tests are fairly self-explanatory. I used the Content and Code website solution because it’s a public site that people can examine if they want to understand more about the structure of the solution and the scope of customisation tested here.

1. Create new NTLM-authenticated web application from the GUI
2. Create new Publishing Portal Site Collection from the GUI, at the root of the new web application
3. Deploy Content and Code website solution from Visual Studio
4. Delete the publishing site collection (this was a necessary step, but not a test that I timed)
5. Create Content and Code website (structure, without content) from the GUI
6. Debug Content and Code website solution in Visual Studio

Core development tests
These tests were added to account for pure development activity for large projects with lots of dependencies. We turned Code Analysis on for the first test because this is a feature that’s very useful but taxes systems pretty heavily. The code deployment times were all fairly small relative to other tests here, but we need to keep in mind that this could be repeated literally hundreds of times per-day. Note: full deployment is accounted for above in the end-to-end test.

• Rebuild Large Project w/Code Analysis
• Deploy Large Project to GAC/BIN

Disk/IO tests
These tests were thrown in because they have an impact on productivity even if they aren’t particularly routine. For the first test I measured the time from turning on the VM until the desktop rendered after logging on. The second test doesn’t really meet the “real world” criteria I name above, but it is a task that can be a productivity barrier in some cases.

• Time to desktop
• Run full crawl (three web apps, no content)

The Testing

The testing process was entirely subject to personal fallibility, as I carried these tests out myself using fairly imprecise methods like a browser-based stopwatch running on my host system (I made sure not to time things inside the guest, where time can slip occasionally). I also went to great lengths to carry out these tests when the systems were performing optimally; I would run through all of the tests once before recording the first set of results. I felt this approach was the best way to discount random variance. The test results were largely very consistent, so I believe these efforts paid off. Obviously the down-side to testing in this manner is that real work is not carried out in a vacuum, but I don’t see any other way to come up with repeatable tests aside from measures like these. It’s what works for science, after all.

The Virtualisation Technologies

As I mentioned in my last post, I chose to limit the virtualisation technologies to a single technology from each of the types I described. I had to postpone testing against “local systems” due to time pressures. It was the option that fell off because we are unlikely to ditch virtualisation any time soon. It works well for us.

To reiterate here, the candidate technologies were VMWare Workstation 7.1, the Hyper-V role in Windows Server 2008 R2 and Amazon’ s EC2 IaaS offering (a Red Hat implementation of the Xen hypervisor). Again, there’s background for all of this in my last post.

What About the Server Room?

One thing I haven’t discussed in any detail so far is VDI or Remote Desktop services. I briefly touched on shared development environments, but I’ve not talked about hosted, individualised development environments. The reason we ruled this out is cost. While this would probably be the best-performing option, all other things being equal, the costs associated with providing this level of performance in the server room would be pretty enormous. For our purposes we might have exceeded power, cooling and weight limitations before we considered the costs of new blade centres and SANs. These costs would probably be even greater in the datacentre. In short, the same criticism applies to individualised hosted development environments as to shared environments: redundancy and resilience at this level is overkill given the associated costs. The data is not critical and anything that needs to be backed up can be stored elsewhere (like TFS).

Basically, people opt for VDI or Remote Desktop services because a mass of underutilised desktop systems can be heavily consolidated. These systems are not underutilised.

The Hardware Candidates

Dell XPS M1330
This is our current laptop model, upgraded with a 320GB 7200 RPM local hard drive and 8GB RAM. One of the serious options we’re considering is a laptop refresh, due to the age and fail rate of the graphics cards and motherboards on these models.

Dell Studio XPS 1645
This was the least expensive decent i7 laptop I could find for testing purposes, and a leading candidate as a replacement laptop. With an £833 (ex-VAT) starting price it could be bumped up to 8GB RAM for a little over £100 more via Crucial. It’s a very heavy laptop and the glossy shell does it no favours, picking up fingerprints within seconds of use. However, it comes wth a 1.6 GHz i7 processor, 500GB 7200 RPM disk standard, eSATA port and HDMI. No USB3. Basically, nothing here was an absolute deal-breaker for us if performance was good.

ASUS V6-P7H55E
This is a barebones system with the following configuration/cost (as priced at scan.co.uk):

• ASUS V6-P7H55E barebones System = £121.67
• Intel i7 870 (8M Cache, 2.93 GHz) = £217.57
• 4GB Corsair XMS3 DDR3 PC3-10666 (1333) Dual Channel – 4x£56.59 = £226.36
• 1TB Seagate Barracuda SATA 3Gb/s, 7200rpm, 32MB Cache, 8.5 ms, NCQ – 3x£41.94 = £125.82
• Adaptec 1220SA PCI-E RAID Card = £46.40
• ASUS 512MB GeForce G 210 DDR2 NVIDIA Graphics Card = £27.71
• Total = £768.58 (VAT-inclusive)

This system is configured with three internal 1TB hard drives and 16GB RAM. We needed to purchase the RAID card because the motherboard does not have an on-board RAID controller. The graphics card was necessary because there are no integrated graphics on desktop i7 processors (although there are for some i3 and i5 models). The disk configuration was variable, as this was one of the test scenarios. The assumption going in was that two disks would be configured in a RAID 0 stripe or a RAID 1 array, depending on performance outcomes. We would only stripe the disks if there was an obvious, significant performance gain. The third disk would be attached to the on-board SATA controller. I will discuss the recommended configuration in more detail later. Also note: the graphics card supports two monitors across any two of the three outputs, but not three concurrently. Finally, the ASUS V7-P7H55E is nearly identical in every respect. We went with the V6 based on availability.

Other laptop models
During preliminary testing we looked at the Lenovo W510, the Dell Precision 6500 and the Alienware M17x among others. All of these models were candidates that we never ruled out, but we didn’t have sufficient time with them to run the entire set of tests. However, these models had a reasonably similar configuration to the Dell Studio XPS 1645 and the Hyper-V tests we ran on these systems yielded similar results to our test model.

Other desktop models
Obviously a barebones system won’t appeal to everyone as a business solution, and it took me some time to persuade myself that it might be suitable for these environments. It wasn’t until I actually priced up this model and compared it to the comparable Dell T1500 (+~£600) and HP Z200 (slower than either model, and pricier) that I considered how it might work for us more seriously.

What am I examining, and not examining?

We have an old laptop, a new laptop, a new desktop and the cloud. Excepting the cloud (which is fixed), we’re permuting each of these hardware options with VMWare Workstation and Hyper-V test results. We’re then adding tests to examine the impact of spindle/bus speeds and the impact of adding/removing cores to these VMs. Ultimately, I wanted to quantify the productivity impacts of a change to our hardware and/or virtualisation technology as opposed to a change within our virtualisation technology, insofar as these tests could be decoupled.

I am not examining every virtualisation solution nor every hardware permutation but I do try to account for a number of these variables with these tests. I would love it if people carried out similar tests on their environments to help build knowledge in an area that’s hugely uninspected today. These are some of the other tests that I hope to revisit next year:

• The impact of application pooling on first page load times. Preliminary tests suggested there might be a small impact, but nowhere near as significant as the User Profile Service Connection. This warrants further inspection.
• The performance of “local systems” on this same hardware. As I mention above, these tests had to be de-prioritised, but I feel it would be worth identifying if there are any of these development-specific tasks where some, or all virtual technologies suffer.
• While I am running tests against a number of disk buses and configurations, I did not get the opportunity to test SSD performance. Obviously a lot of people will want to know the impact of SSD on these timings, but unfortunately I won’t have an opportunity to inspect that until early next year at the earliest.
• In some cases we work with deep snapshot trees. I want to gain an understanding of how differencing across ten or more files impacts performance for these tasks.
• Compare performance of a higher-clocked i5 to a lower-clocked i7 at a similar price range and potentially explore over-clocking options.
• Compare slower memory on an otherwise-identical system.
• Run VirtualBox tests on an otherwise-identical system.
• Assess the impact of virtualisation optimisations.

Obviously these tests say nothing about the usability of the system, power costs, mobility and more. For the purposes of this post I’m only concerned with outlining how I tested system performance for these real world tasks. In the next post, at long last, I will share the results.

For clarity, I will quickly state the problem as I see it. SharePoint 2010 system requirements and practitioner mobility requirements are inherently at odds. What guidance exists for this unique problem space tends to regurgitate preferences/allegiances rather than comparing technologies and ratifying assumptions with real-world tests. At best, you get system performance indices for a single laptop model, but these results may vary when any hardware component is changed.

How can virtualisation improve system performance?

It doesn’t. People look to virtualisation to solve other problems. However, SharePoint 2010 performs differently in different virtualisation technologies, and the margins of these differences vary by hardware configuration. By all means, the advantages of virtualisation often make it a desirable choice, but these performance characteristics need to be accounted for, lest system performance losses negate the productivity improvements that virtualisation can introduce.

Why virtualise?

There are a number of advantages to virtual systems over physical systems. Many of these benefits can also be obtained with sufficiently mature systems management technologies and physical systems, but these benefits are often easier, quicker or less costly to implement through virtualisation. Some of the benefits include:

• Provisioning times for new SharePoint environments.
• Standardisation through cloned, network-isolated virtual machines.
• Account for volatility with snapshots.
• Standard builds per-project, to share with team members, reducing project initiation costs.
• Virtual appliances produced by Microsoft and third parties, such as the Information Worker Demo VM.
• Reduced hardware rebuilds by removing development tools and SharePoint from the host.

This list is by no means comprehensive. As I say, many of these benefits can be realised with scripting and/or management tools. This list is only intended to illustrate why it’s a powerful design option.

An overview of virtualisation and related technologies

Some example technologies by type:

• Type I Hypervisors
  • VMWare ESXi
  • Hyper-V
• Type II Hypervisors
  • Oracle VirtualBox
  • VMWare Workstation
• Infrastructure as a Service (IaaS)
  • Amazon EC2
  • Azure VM Role (forthcoming)
• Local Systems
  • Native Boot Windows 7 (virtual hard disk)
  • Citrix XenDesktop (VDI)

Note: Virtual PC was not included because it doesn’t support 64-bit guest operating systems. SharePoint 2010 only runs on 64-bit systems.

Some of the alleged benefits of these approaches:

• Type I Hypervisors
  • Better performance**
  • Good management options/tools
• Type II Hypervisors
  • Host Operating System
  • Easy to use
• Infrastructure as a Service (IaaS)
  • Pay as you go
  • Scalability
• Local Systems
  • Good performance
  • Simple to use

Some of the alleged drawbacks of these approaches:

• Type I Hypervisors
  • No Host Operating System***
  • Driver issues*
  • Complicated
• Type II Hypervisors
  • Historically poor performance**
  • Historically, less manageable (snapshots, import/export, etc)
• Infrastructure as a Service (IaaS)
  • Requires stable connectivity
  • Complicated
  • Pay-As-You-Go requires diligence
• Local Systems
  • Easy to damage
  • Slow to rebuild

*Hyper-V has driver issues on some newer laptops. These are most noticeable with graphics, although I have seen audio driver problems as well. Some of these driver issues may be fixed or alleviated in the SP1 Beta/RC for Windows Server 2008 R2.

**This performance bias is one of the things I will be examining in more detail in later posts.

***This is only “sort of” true for Hyper-V, which invokes a “parent partition”. This is a special type of virtual machine that fulfils a similar role to a host operating system, and is often referred to as such.

Why are “Local Systems” included?

I’ve lumped these in for two reasons. 1) They share some characteristics with the other virtualisation technologies, like running from virtual hard drives. 2) By virtue of being local systems, they fundamentally negate some of the benefits that are obtained through virtualisation. Cloning these machines is not an option if SharePoint is installed and configured. It will be necessary to invest in scripting environment provision in order to retain those productivity benefits. It happens that many people choose to take this scripting approach, but it’s worth pointing out that network isolation and cloning can achieve similar results through virtualisation, and this does not obtain with Local Systems.

What about shared, hosted development environments?

In this scenario I’m thinking of hosted development farms, where some or all members of a team use a single environment. Based on my subjective reading of the community, this option seems to be fading away. I think there are three reasons why.

1. Cost. Running development environments on proper infrastructure is expensive. Most components have been made redundant, the storage will be expensive if it performs well, the power/cooling costs are considerably more expensive than for laptops/desktops and you will need to pay people to manage the systems. Even when these costs are split across multiple developers, it’s still expensive unless resources are overcommitted, which negates productivity gains. It also tends to be more expensive to provision new environments and this process can often be an obstacle to business agility. In a nutshell, these are protections that are unnecessary for development environments. Redundancy and resilience at this level is overkill given the associated costs. The most important assets, such as code, standard images and project-specific builds can be protected separately.
2. Hive pollution. If these farms will support multiple projects, as they often do per the previous comments about provisioning, then these systems will inherently differ from the test/stage/UAT/production systems they should resemble. Core files in the hive can be altered from project-to-project, resulting in unexpected behaviour when moving code between these environments. This can seriously complicate troubleshooting and should be avoided.
3. Mobility. These farms aren’t terribly useful to people who are travelling or who are working on-site with restricted outbound connectivity.

All of this said, there are times when project-specific requirements may make shared farms a good option. It may be sensible to take another look for:

• Integration projects.
• Developing with large amounts of data.
• Projects with heavy infrastructure requirements, such as FAST.
  • Perhaps individual development environments can consume a shared FAST Service Application?

Generally speaking, I believe these resources should be provided only in these niche cases.

How is this different from IaaS?

The main differences are costs and capital. Cloud-based infrastructure services are fundamentally just virtualised hosting on an enormous scale. This scale lowers costs to a point where it may be affordable to deploy individual machines per-developer. Although in my analyses I found that IaaS would be more expensive than desktop workstations over three years, this still may be compelling when cash flow issues preclude significant one-time investment or credit flows are restricted. IaaS should also be kept in mind when specific projects require significant provisioning or investment for a short term, for instance testing in a large farm.

While providing a single cloud-based VM per-user solves the first two issues with shared development environments, mobility is still an issue. In many places, stable mobile broadband is flaky at best. Additionally, there are key architectural differences that need to be accounted for when working in the cloud, and on a Pay-As-You-Go basis. I address all of this in my series on SharePoint 2010 Infrastructure for Amazon EC2.

Which approach is best?

This is a high-level overview of the design constraints that limited my choices, before I plunged into a concrete performance review of the remaining technologies.

Local Systems
In my view, Local Systems are only a better choice if the supporting IT systems and processes are very mature and the performance benefits are clear and significant. For most development scenarios, that has yet to be proven. I’ve postponed this virtual to physical performance comparison for now, as the other benefits of virtualisation have ruled this approach out for us, but I hope to revisit it in the new year.

IaaS
IaaS has two key planning considerations. The first is fairly obvious. Outbound RDP Connectivity needs to be open whenever the systems are needed. I encourage people to consider this in some detail and pilot with many types of users before diving in. The second consideration is Pay-As-You-Go. While cloud providers often have an always-on option, it’s usually pretty pricey. The alternative is to find a mechanism to limit compute usage to when it is truly being used, without introducing usability problems. Management tools or scripting may be able to answer these problems, but no one should enter in to this process thinking it will be easy. This is not an easy option. For a more detailed consideration of these issues, refer to my series on EC2.

Type II Hypervisors
VMWare Workstation is the most mature desktop virtualisation product on the market, although in recent years VirtualBox has been gaining share. Choosing between these technologies for my tests was never going to be easy, but I reduced it to a few factors:

• I’ve never met a VirtualBox user that would complain about using VMWare but I can’t say that proposition is reversible. There are a lot of SharePoint practitioners with a strong preference for VMWare.
• VMWare Workstation has native interoperability with other VMWare assets. While VirtualBox supports the VMDK file format, it’s not quite the same thing.
• Both products are fairly inexpensive in the grand scheme of things.
• I had stability issues with VirtualBox circa version 3.14 that left a bad taste in my mouth.

Perhaps most importantly, I felt that the performance comparison of VMWare Workstation to Hyper-V would be the most valuable decision-making information.

Type I Hypervisors
Most Type I Hypervisors would not be suitable for desktop virtualisation because they don’t have a host operating system. While it would be possible to boot a guest OS and remote in to other Virtual Machines over internal networks, this is a complicated approach and the networking requirements would be enough to put off most developers. However, as mentioned above, Hyper-V is a notable pseudo-exception to this with its parent partition.

We’ve been using the Hyper-V role in Windows Server 2008 R2 for development for a little over a year now. While we have successfully capitalised on many of the productivity benefits of virtualisation through this approach, there are a few issues that have never been entirely satisfactory:

• Despite having the host OS, using Hyper-V is still complicated for non-Systems people – particularly the networking.
  • Work-around solutions for Wireless networking are fiddly.
  • Lack of self-contained NAT requires the use of Internet Connection Sharing in order to achieve network isolation, which some users struggle with.
• Lack of Sleep/Hibernate is painful for many users.
• Graphics performance is poor – particularly with large PowerPoint/Visio files, large images and video.
• Audio can also suffer during large file operations.
• Hyper-V is not ready for laptop power schemes.

Despite these niggles, we’ve continued to use Hyper-V while waiting for the forthcoming graphics/memory improvements in Windows Server 2008 R2 SP1. I would class these usability problems as significant inconveniences that sometimes manifest themselves in lost productivity – particularly with new users learning our approach.

New Problems in SharePoint 2010

Since we properly immersed ourselves in SharePoint 2010 development, negative reports about performance started to roll in. These proved hard to validate until a few months ago when my colleagues showed me first page load times after an IISRESET in excess of one minute. This was concrete and repeatable. The problem was more severe on some systems than others, but it was clearly a problem.

The performance tests I’ve been conducting have been an effort to pick apart these results in Hyper-V. Was this new in SharePoint 2010 or did it amplify something that was minor before? Do we get the same problems on different virtualisation technologies, in the cloud or is this a symptom of virtualisation itself? In my next post I’ll discuss the environments, the tests and the testing process.

I hit the search engines in earnest and found that these problems were prevalent across a fairly wide range of graphics cards. We enlisted Dell’s help and they told us that they do not certify that Hyper-V will work on any laptops. More precisely, they clarified the primary support concern is that future driver releases may not work with Hyper-V even if we find a model that works with today’s drivers. At this point we were considering a pricier Precision model and they put us in touch with their Precision product team in Texas. They were most helpful but we were told that Dell themselves do not use Hyper-V on laptops except for demonstration purposes and they simply use it as a server for connected workstations, so they would never experience the same graphics issues. Dell kindly offered to let us test our development build on various models at their campus if we agreed to share the results with them, but before we could arrange that visit, Windows Server 2008 R2 SP1 Beta* was released and I upgraded my machine in order to test out Dynamic Memory.

As I was installing it I had a chat with my colleague (and serial early adopter) Lambros Vasiliou to gauge his impressions. He mentioned his favourite improvement is that the known Hyper-V host graphics performance issues are either gone or greatly mitigated. This is an issue that’s been repeatedly discussed in our organisation since we moved from a hotchpotch of virtualisation technologies to Hyper-V as our standard development build last year. It’s probably the single thing that irritates our users of this system more than anything else.

I did some testing myself with videos playing and moving windows about with Windows Key + Arrow hot keys. The results were fairly impressive – without doubt a big improvement. One thing that still behaved poorly on my Dell XPS M1330 (with NVIDIA GeForce 8400GS) is full-screen YouTube, Vimeo, etc. The CTRL+ALT+DEL redraw operation seems a bit sluggish still as well. I noticed that my PowerPoint Presenter View was better, but still not 100% responsive.

I also tested on the Dell Latitude E6410 (with NVIDIA NVS 3100M). Not only is the previously-mentioned blue screen fixed and the graphics generally improved in the same ways as on the XPS, but the full-screen in-browser video and CTRL+ALT+DEL are instantaneous. One possible explanation for this different experience is that the Latitude has a processor with SLAT, but I can’t validate that at all yet… because I can’t find any information whatsoever about why/how this has changed!

I think it’s unlikely that these changes are related to RemoteFX (since the XPS M1330 does not have a processor with SLAT and I never enabled it on the Latitude E6410). I would expect RemoteFX to improve the experience connecting to the guests, not the Hyper-V root partition (although it’s possible that this improvement is somehow related). I’ve tried pinging Virtual PC Guy and posted this query on the SP1 Beta TechNet forum but so far the community can only confirm that this is indeed working on a number of different models including a Mac (drill down in the links on the TechNet thread for more information). One way or the other this is great news, but I’m finding the lack of information about these changes quite maddening given the amazing detail that’s been produced for the Dynamic Memory launch. I’d really appreciate further insights if anyone can reveal the internals.

* A few notes regarding the Service Pack 1 Beta installation process:

1. The links on the SP1 Beta page are a bit confusing. You should be aware that if you click the “Evaluate Windows Server 2008 R2 and SP1 Beta” link you will be taken to a page with a “Download Windows Server 2008 R2 Trial Software” section at the top. “Download SP1 Beta Software” is beneath that section. This is what you want. If you click the first link you will initiate a download of the full Windows Server 2008 R2 (SP0) installer. If you “upgrade” your system using that installer you’ll wind up with a nice new trial version of SP0. AGH. Starting again from the links in the right section I was able to run a small installer that presents the updates to Windows Update and that has all worked fine, so I’d recommend that route. Alternately the Service Pack can be downloaded stand-alone. I did that for my second install and it worked fine too. Also note the Windows Server 2008 R2 SP1 Beta Reviewer’s Guide, “to evaluate the core features of Windows Server 2008 R2 SP1 Beta release in your environment”.
2. If you use Forefront you will need to uninstall it in order to install SP1 Beta, so make sure to remember to reinstall it afterwards.
3. When I installed the Service Pack my screen went black for about ten minutes following the first reboot. Be prepared for this. You’ll see plenty of ongoing disk activity but nothing on the screen. Fairly disconcerting, but perhaps this is all a part of these same video changes.

Exchange Server Reconfiguration

Tidying up the Exchange server is a much more straight-forward process. In fact, all of the changes that I made are network orientated per the network changes from the first post, so if you are not adding a second NIC or a second fixed IP address on the original internal NIC, these steps aren’t necessary.

• Import the virtual machine. Plug the Hyper-V Internal Network in to the first NIC and add a second NIC with the Hyper-V ICS Network plugged in to it.
• The initial IP address is 192.168.150.2.
• Rename the Local Area Connection NIC to Hyper-V Internal Connection (or your preference).
• Rename the Local Area Connection 2 (or maybe 3) NIC to Hyper-V ICS Connection (or your preference).
• In IPv4 properties on the Hyper-V ICS Connection, change the Advanced TCP/IP Settings to not “Register this connection’s addresses in DNS”, as it is dynamic.
• I disabled IPv6 on both NICs as it is already disable on our host’s network connections.
• Add a second IPv4 address to the Hyper-V Internal Connection:  192.168.200.151/255.255.255.0.
  • Added the same address on the DNS tab for this NIC.

• Check DNS to make sure the new address was added.
• Added a HOSTS file entry for demo2010b pointing at 192.168.200.151.
• Tested logging on via Remote Desktop from the Host machine.

Shutdown/snapshot and optionally export the VM if time/resources permit.

Reviewing SharePoint Server event logs

This section details my review of the event logs in the shipped state. I did not take any action except for the last two items regarding DCOM fixes and the SharePoint Health Logs. I believe that most of these errors are probably an effect of running so many things in one environment, but I’d welcome comments if you have any insight to share.

• SetSPN for WSMAN warnings: The WinRM service failed to create the following SPNs: WSMAN/demo2010a.contoso.com; WSMAN/demo2010a.
  Additional Data
  The error received was 8344: %%8344.
  User Action
  The SPNs can be created by an administrator using setspn.exe utility.
  Presumably these SPNs can be created manually or the rights to create the SPNs can be assigned to the WinRM service account if needed, but I am not making any changes here until I see that it is necessary to do so. There are some Kerberos Audit Failures in the security logs but since the SharePoint environment is self-contained and there are no secondary hops, I don’t think this is worthwhile.
• VSS Error 8320: Volume Shadow Copy Service error: Failed resolving account Administrator with status 1376. Check connection to domain controller and VssAccessControl registry key.
  Operation:
  Initializing Writer
  Context:
  Writer Class Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
  Writer Name: WMI Writer
  Error-specific details:
  Error: NetLocalGroupGetMemebers(Administrator), 0×80070560, The specified local group does not exist.
  There are a number of articles that discuss fixes for this Warning but since this is an unusual configuration (SharePoint/SQL on a DC) and the warning is about the “Administrator” account, I am hesitant to make this change, for fear of introducing instability. Further reading at Experts Exchange, Technet and AnandTech.
• There are two errors regarding the ULS config file, 7105 and 7056. I’m honestly unsure what to make of these errors since they refer to C:\Program Files\Common Files\Microsoft Shared\ULS\14\uls.config.xml and I’m not certain how that is used in generating the ULS log files for SharePoint which are appearing normally at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\LOGS. Not taking any action for now.
• Microsoft.ResourceManagement.ServiceHealthSource Error 22:
  The Forefront Identity Manager Service cannot connect to the SQL Database Server.
  The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.
  Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, verify the connection information has been configured properly. This configuration is stored in the Windows Registry.
  This error seems to occurs before the FIM Sync service starts, which may or may not be a related issue. The User Profile Service Application is available though, so I believe this has something to do with this connection attempt occurring before the Service Application is able to connect with SQL.
• CAPI2 Error:
  Failed extract of third-party root list from auto update cab at: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab> with error: A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.
  This issue seems to have emerged in July of this year on Windows 7 and Windows Server 2008 R2 machines. Unfortunately there’s no clear resolution at this time, but also no clear negative impact on this system, so I’m not taking any action for now.
• DCOM 10016 error. A fix for this has been documented well by Matt Groves.
• SharePoint Health Reports:
  • Almost all of the warnings and errors in the Health Report are either one-time issues that no longer exist or are symptoms of the single-server install on a DC.
  • An exception to this is the “Validate the My Site Host and individual My Sites are on a dedicated Web application and separate URL domain” message, which is accurate. The environment has not been configured with a dedicated MySite application.
  • The one other issue was “The Unattended Service Account Application ID is not specified or has an invalid value”. This has not been set up in the VM and will need to be configured in the Secure Store Service if it will be used.
  • I have chosen to disable all of these rules, as they are undesirable in a demo environment. I also deleted the existing alerts.

Pending

As I mentioned above, I hope to revisit this VM and create new snapshots that will reduce load by disabling bulky services. This snapshot branch might resemble something like this:

• Current state, as above
  • Most SharePoint Services turned off in Services on Server
  • BI Indexing Connector Service , FAST, OCS and Project Server removed. All SharePoint Services turned on
  • As above, but with FAST
  • As above, but with OCS
  • As above, but with Project Server
  • BI Indexing Connector Service , FAST, OCS and Project Server removed. All SharePoint Services turned off
  • As above, but with Search/FAST
  • As above, but with OCS
  • As above, but with Project Server

I really haven’t planned this yet or discussed these options with users of this environment, so it could wind up looking completely different. Also, this many snapshots would probably chew up too much disk space and get confusing for the users. But watch this space for updates as I hope to revisit the topic again. And please feel free to suggest other optimisations or additions that work well for you.

Now the public beta trial is expiring and people are moving to the RTM build. It appears to be much improved, in that more of the product works in this version and a few niggles have been fixed now. However, it’s widely acknowledged that the resource requirements for this virtual machine are gargantuan due to the breadth of what it offers.

What you get

The first of these VMs doesn’t just have SharePoint Server 2010 (Enterprise). It has:

• Active Directory Domain Services
• DNS
• SQL 2008 R2
• Office Professional Plus 2010 (including Visio and Project)
• SharePoint Designer 2010
• Visual Studio 2010
• Office Web Applications
• FAST Search for SharePoint 2010
• Project Server 2010
• Office Communication Server 2007 R2

There is a second VM with Exchange Server 2010 which you can use as needed. Note: when downloading the RAR files which self-extract to the VM export files, I highly recommend using the Akamai download links. They will save you time and frustration.

Why I’m writing about it

The first of these VMs does a lot, and will chew up more virtual resources than most physical demo machines have to provide. In this post I documented the tweaks I made to our copy of the Virtual Machines and I’ve reviewed the event logs to identify any other issues that are present in the shipped state. I also spent some time fixing the start-up of some services due to observed delays in network connectivity.

The optimisations achieved here are not immense but they establish a firmer baseline from which to carry out further improvements. Unfortunately I have not had the time to explore the best approach to taking these core assets and removing the bulky bits in order to provide snapshots that unleash bits of the functionality as needed. With time I hope to revisit this and potentially explore other uses for this environment.

SharePoint Server Reconfiguration

These are the steps I’ve taken to reconfigure the SharePoint Virtual Machine:

• Imported the virtual machine. Plugged the Hyper-V Internal Network in to the first NIC and added a second NIC with the Hyper-V ICS Network plugged in to it. To understand more about the NIC naming conventions I’m using here and how we use these Hyper-V networks, please refer to my series on SharePoint development environments.
  • If you’re curious, we dual-boot our Sales laptops in to this development environment so our sales people can use Hyper-V as needed.
• The default SharePoint VM RAM is set to 5120MB (5GB) RAM. This could be increased up to ~6GB on a host with 8GB RAM, assuming the load on the root partition is minimal during demonstration and the Exchange VM is not being used concurrently.
• If using a UK keyboard through the Hyper-V Virtual Machine Connection, the password will need to be entered as pass“word1
• Note: at login a warm-up script runs, which I will discuss more in a moment.
• I changed the time zone to London.
• I changed UAC to “Notify me only when programs try to make change to my computer (do not dim my desktop)”. I find that with Known Hyper-V graphics performance issues this setting achieves the right balance between usability and security.

Service start-up

• I reviewed the event logs to identify which Manual Services would start up at a delay. Approximately five minutes after the machine started the Virtual Disk Service started up. That was followed by the following services:

  Service	Startup Type
  Application Experience	Manual
  BI Indexing Connector Service	Automatic (Delayed Start)
  Certificate Propagation	Manual
  Diagnostic Policy Service	Automatic (Delayed Start)
  Diagnostic System Host	Manual
  Distributed Transaction Coordinator	Automatic (Delayed Start)
  FAST Search for SharePoint	Automatic
  FAST Search for SharePoint Browser Engine	Manual
  FAST Search for SharePoint QRProxy	Manual
  FAST Search for SharePoint Sam Admin	Manual
  FAST Search for SharePoint Sam Worker	Manual
  Function Discovery Provider Host	Manual
  IPsec Policy Agent	Manual
  Microsoft .Net Framework NGEN v4.0.30319_X86	Automatic (Delayed Start)
  Microsoft .Net Framework NGEN v4.0.30319_X64	Automatic (Delayed Start)
  Network Connections	Manual
  Network List Service	Manual
  Office 64 Source Engine	Manual
  Office Software Protection Platform	Manual
  Portable Device Enumerator Service	Manual
  Remote Access Connection Manager	Manual
  Remote Desktop Services	Manual
  Remote Desktop Services Configuration	Manual
  Remote Desktop Services UserMode Port Redirector	Manual
  Secure Socket Tunneling Protocol Service	Manual
  SPP Notification Service	Manual
  SQL Full-Text Daemon Launcher	Manual
  Telephony	Manual
  Windows Defender	Automatic (Delayed Start)
  Windows Module Installer	Manual
  Windows Remote Management	Automatic (Delayed Start)
  Windows Search	Automatic (Delayed Start)
  Windows Update	Automatic (Delayed Start)
  WinHTTP Web Proxy Auto-Discovery Service	Manual

• In my testing, switching the Network Connections and WinHTTP Web Proxy Auto-Discovery Service services to Automatic would reduce time to CTRL+ALT+DEL from five minutes to three minutes. It’s worth verifying these results in your environment.

Service Hardening

I disable the following services, per my normal virtual machine hardening processes. I don’t believe that any of them are necessary in this environment:

• Certificate Propagation
• Desktop Windows Manager Session Manager
• Distributed Link Tracking Client
• Encrypted File System
• Function Discovery Provider Host
• Function Discovery Resource Publication
• IP Helper
• Microsoft iSCSI Initiator Service
• Multimedia Class Scheduler
• Problem Reports and Solutions Control Panel Support
• Remote Procedure Call (RPC) Locator
• Smart Card
• Smart Card Removal Policy
• Special Administration Console Helper
• Tablet PC Input Service
• Windows Audio
• Windows Audio Endpoint Builder
• Windows Error Reporting Service
• Windows Search
• Wired AutoConfig

I also disabled these services. I have no idea why they were ever enabled.

• Microsoft Fibre Channel Platform Registration Service
• Microsoft FTP Service

Following a reboot to assess boot times after hardening, I was prompted with a firewall exception warning for SharePoint Workspace. I allowed this exception.

MSCONFIG

I mentioned earlier that a warm-up script launched at login. I reviewed the contents of the script and identified that it enumerated all sites in the farm, so I browsed to each site that was getting warmed up to have a look at that content. Most of these sites pull up the “select a template” page, indicating that there is not content in the web application, so I have chosen to disable the script for now and will likely revisit the use of the IIS.NET application warm-up module when it matures. In the mean time, I recommend that recipients of this build take a snapshot of the VM while the http://intranet and Central Admin web applications are fully warmed up for their needs so they can roll back to that point as needed.

While in MCSONFIG, I turned off:

• Watson Subscriber for SENS Network Notifications
• EcmCopyLinks
• Warm Up
• Optional:
  • Microsoft Office 2010 (these are sync services – probably best to leave them running)
  • Microsoft Office Communicator 2007 R2 (this is preferential)

Housekeeping and simple benchmarks

At this point I wanted to take some basic start-up timings so I shut down the VM, but before starting back up I changed the Hyper-V VM’s BIOS settings to boot from IDE before CD. After that I took some pretty unscientific measurements of boot times and resource consumption on my Dell XPS M1330:

• CTRL+ALT+DEL: 2:56.
• Desktop: 3:47.
• Baseline RAM consumption down from >4.5GB to ~3.65GB before hitting the intranet and Central Administration web apps (remember, they aren’t warmed up anymore).
• RAM consumption at just under 4GB after hitting Central Administration.
• RAM consumption at ~4.25GB after hitting http://intranet.

After taking these measurements I shut down the VM and took a snapshot.

Network changes

These changes are optional, depending on how you will use the VM in your environment. I opted to add a second IP address on our existing internal network range, as follows:

• The initial IP address is 192.168.150.1/255.255.255.0.
• Rename the Local Area Connection NIC to Hyper-V Internal Connection (or your preference).
• Rename the Local Area Connection 2 (or maybe 3) NIC to Hyper-V ICS Connection (or your preference).
• In IPv4 properties on the Hyper-V ICS Connection, change the Advanced TCP/IP Settings to not “Register this connection’s addresses in DNS”, as it is dynamic.
• I disabled IPv6 on both NICs as it is already disable on our host’s network connections.
• Add a second IPv4 address to the Hyper-V Internal Connection: 192.168.200.150/255.255.255.0.
  • I also added 192.168.200.150 on the DNS tab for this NIC.
• Check DNS to make sure the new address is added.
• Add a HOSTS file entry in the root partition for demo2010a pointing at 192.168.200.150.
  • If desirable, add another entry pointing for intranet.contoso.com pointing at 192.168.200.150. This can be used for browsing from the root partition’s browser.
• Tested logging on via Remote Desktop and browsing to Central Admin from the Host machine.

Client tools and other changes

Some final changes are optional, but I think generally desirable.

• Install Firefox, Opera, Safari, Chrome.
• Install PDF Exchange Viewer or your choice of PDF viewer.
• Update: As mentioned by Leon Zandman in the comments here, the free SysInternals ZoomIt tool is very useful for presentations. You may also want to consider adding it.
• Add the environment variable path to the 14 hive’s BIN.

And that’s it for the SharePoint VM. Shutdown and take a new snapshot, optionally deleting the first one. I would suggest exporting the VM if time/resources permit, noting that export operations can be time-consuming and disk intensive.

In part two I’ll discuss the Exchange VM, event logs and potential future improvements.

• Microsoft’s approach to Dynamic Memory is fundamentally different than VMWare’s overcommitment, in that VMWare doesn’t trust information about memory usage from within the guest, whereas Microsoft’s implementation is based around an awareness of the amount and type of memory that’s being used at all times.
• Dynamic Memory will work by Adding/Removing memory.
  • Adding memory is enabled through a new synthetic memory driver.
  • Removing memory that’s not being used is done with ballooning.
  • Memory is now assigned with a few new values:
    • Startup memory is the amount of memory assigned to a VM, which is also the minimum memory the VM will consume (default value is 512 MB).
    • Maximum memory limits how much memory a VM can consume.
    • Priority can be assigned to specific VMs in order to make sure that they receive available memory before other lower-priority VMs.
    • A Memory Buffer can be set to reserve memory for specific VMs, for instance if they need extra memory for file caching.
• Hyper-V Manager adds two new columns.
  • Current Memory identifies how much memory the VM is consuming.
  • Memory Availability identifies the difference between how much memory a VM has vs. wants in a +/-% figure.
    • When the availability goes negative, the Windows guest will start to work with the lesser amount of memory that’s now available to it (via paging, etc).
    • Negative availability will result in reduced performance, but the systems will continue to function.
• Memory is now reserved for the root partition in a different way, so that dynamic memory won’t bring down the host.
  • This amount can be configured with a new registry key based on how the root partition is being used, for instance if it’s your desktop OS.
• As Dynamic Memory is used more, the chances of spanning NUMA nodes increases (on NUMA systems).
  • He points out that different systems have vastly different Back Channel performance, so the impact of NUMA Spanning can be negligible or drastic.
  • In SP1, NUMA Spanning can be disabled (if desired).
• Dynamic Memory also supports Large Pages, which are likely to become more common with virtualised Exchange/SQL.
  • VMWare cannot overcommit these pages.
• I’ve asked if there are specific processor requirements. I’ll be interested to see how/if this supports processors that don’t have SLAT.

After that I tried to delete other snapshots but I kept getting errors and the VM entered a Saved-Critical state. This will happen when Hyper-V Manager cannot access a file system location or cannot find a file, for instance when a removable hard drive is pulled out.

Approximately 30 seconds later, Hyper-V thought that it regained access to the location:

However, I couldn’t get any snapshots to delete and the virtual machine wouldn’t start. After a few minutes of panicked clicking I decided to restart the Hyper-V services. When they came back up my VM disappeared. The Virtual Machine configuration file was corrupted.

The next event suggests that a snapshot file was also corrupted.

These 16310 and 16330 errors repeated for a while. Panic continued. Eventually I rebooted. On reboot the VM was still missing and the 16310/16330 errors persisted.

On a hunch I decided to see what the AVHD files (the differencing disks that correlate with snapshot states) looked like.

This looked very much like what I would have expected if nothing had gone wrong (and if none of the snapshot deletions completed). Sticking with this line of inquiry (and what the 16310 error suggests), I created a new virtual machine and pointed it at the most recent AVHD file (selected above). All of my snapshots were missing but the virtual machine created successfully. I started the virtual machine and it was clearly in the same state it was in before I took the most recent snapshot, with a few caveats. In my panic I forgot to re-create my second NIC, so the VM started with only one (the one that I specified when I created the new VM). I also forgot to give it a second CPU. So I shut down the VM, made these changes, restarted, reconfigured the second NIC and tested that everything worked to my expectations. Recovery complete, so I shut down both VMs again.

At this point I’d recovered the VM but I still had a bunch of unnecessary data in my branch of differencing disks. In order to clean this up, I took a new snapshot of both VMs and exported the latest snapshot of each of them. This merged all the differences across the AVHD files in to a new, self-contained VHD file. After the exports finished I deleted the old VMs, waited for the Destroy operations to complete, cleaned up lingering files on the file system and imported the new exports. I took a new snapshot, as this is my new stable starting point and everything was (relatively speaking) back to normal. Phew!

With hindsight, I would have handled the recovery as follows:

• Create new VM, pointing at latest differencing disk (or whichever snapshot state you wanted to preserve).
• Reconfigure processors.
• Reconfigure network adapters in Hyper-V Manager.
• Start the virtual machine.
• Reconfigure NICs in the guest.
• Reboot.
• Test everything is working as expected in the VM.
• Shut Down.
• Snapshot.
• Export.
• Delete old VM from Hyper-V Manager.
• Wait for the Destroy operation to complete.
• Delete any lingering files from the file system.
• Import the exported virtual machines.

As I hinted at above, having gone through this process, it occurred to me that you could probably point at whichever AVHD file you wanted to, if you didn’t want to use the latest snapshot, assuming none of the AVHD files were corrupted. In this case it was just the virtual machine XML file and possibly the snapshot file that were corrupted, rather than the VHD file and differencing disks (AVHD files) themselves. The problem would be identifying which AVHD file corresponds to the snapshot that you want to keep, but in principal I think this would work.

I should note that this is probably unsupported, but you’re not really losing anything because otherwise you would have only been able to recover the first VHD file. This technique wouldn’t be much use if you didn’t know which snapshot you were after or if you wanted to recover the entire snapshot tree, but this fix gives you some recovery where the virtual machine file and the snapshot tree are corrupted but the disk data is not.

Cloning isolated VMs vs. scripted installation

One of the challenges we’ve always faced with SharePoint development has been the tension between cloning actually identical environments versus automating the deployment across distinct environments (or worse, repeating the installation manually). In the first case we save time by eliminating reconfiguration and this ensures a consistent experience for each user. This is particularly beneficial for software development. These benefits can also be obtained by scripting installation/configuration/deployment but there’s a considerable overhead associated with developing and testing those scripts. As SharePoint 2010 is still quite new and we’ve been working on projects for some time now, we didn’t have the luxury of waiting for those refinements and we needed to take advantage of these efficiencies as we had done with SharePoint 2007 projects.

This cloning approach has one big drawback: it requires network isolation in order to prevent network, machine or SID duplication issues. We can make up for that partially by using Network Address Translation (NAT) so that we can get outbound connectivity from the VMs (we do this using Internet Connection Sharing to an internal network), but this doesn’t provide any inbound connectivity for Remote Desktop connections to the VMs or the SharePoint applications in them. Remote Desktop access may be desirable because of the limitations of the Hyper-V Virtual Machine Connection or because the users do not have permissions to Hyper-V itself. Alternately, it may be desirable to expose the virtual machine’s browser or another application directly, as I’ll describe later in this article.

Before I detail the approach to publishing a network-isolated remote desktop connection, I should note that it’s also possible to lock down the environment using Windows Firewall and Hyper-V networking so that inbound traffic is isolated enough that most duplication issues are accounted for – but this requires considerable planning, additional reconfiguration and it’s a large enough topic that it requires another post (I will try to revisit this soon). In short, I think each approach addresses different needs. For now, I’ll talk about RemoteApp.

RemoteApp to a Remote Desktop

It’s been some time since I looked at the new features in Windows Server 2008 R2. I spent most of last Summer getting to grips with them but have been living in a world of SharePoint 2010 since, so I never got a chance to revisit the interesting developments with Remote Desktop Services. Among them is RemoteApp, which allows us to publish applications (rather than a terminal session) on a server through Remote Desktop Web Access. That application could be Word, PhotoShop, SQL Server Management Studio, ULS Viewer or whatever might be useful.

In the last couple of weeks I’ve addressed a couple of requirements by publishing a saved Remote Desktop connection file through Remote Desktop Web Access. To give a brief overview of the solution, I’ve logged on to the Hyper-V host server and saved a Remote Desktop connection file pointed at the target Virtual Machine. This relies on the Hyper-V host’s internal network connection to the virtual machine and our LAN connection to the Hyper-V host (the VM does not have an external network connection). Then I published that saved remote desktop connection file using RemoteApp. Users launch the connection through the Remote Desktop Web Application. This is a very basic example of how RemoteApp can help us by bridging a network barrier that we have to maintain.

RemoteApp to a Remote Application

If we get slightly more complex we can also get to the real beauty of this approach. RDP connection files can now be customised so that you can expose an application inside of the target machine. This is like Windows 7′s XP Mode, except I’m able to take advantage of the remote machine’s applications through Remote Desktop Services without having to log on to either the Hyper-V server’s terminal or the virtual machine. Put another way, I can log on to my Hyper-V server’s Remote Desktop Web Application portal and launch an RDP file. That RDP file bypasses both terminals and provides me with the application that I’m really after – say, ULS Viewer, or a browser that can render the network-isolated SharePoint data. Here’s a more detailed scenario:

• A Hyper-V server on the network has a number of virtual machines that fully replicate our domain infrastructure.
  • Because this is an exact copy of our live environment we need to keep the test environment completely network-isolated.
• We’re testing Word 2010 on multiple operating systems before rolling it out across the network.
• The tester needs to be able to access the Word Application but should not have access to the rest of the infrastructure.
• We have XP, Vista and Windows 7 VMs on the Hyper-V server – all running Word 2010.
• We log on to the Hyper-V server and create saved RDP files to each of those VMs.
  • We modify those RDP files so that they expose only the Word application. This is a few lines of added or modified XML in the connection file and a change to the TsAppAllowList registry key on each virtual machine. More detail on that approach here.
• We add each of these saved RDP files as Remote Applications in the RemoteApp Manager Remote Desktop Services node on the Hyper-V server. To be clear, this is new Remote Desktop Services functionality. It only happens to be a Hyper-V server because we’re exposing Hyper-V guests.
• The tester logs on to the Remote Desktop Web Access site and launches the RDP files as needed. Each file will expose an instance of Word 2010 which is running in the remote (and isolated) virtual machine’s context.

This experience is improved if you install the Hyper-V server’s certificate for the Remote Desktop Web Application site. Also note, you will be prompted twice to authenticate – once to the Hyper-V server and once to the virtual machine. That’s not a great user experience, but it’s also not the worst. To be honest, I haven’t had the time to see if it can be improved with Kerberos or another approach. I’d be interested to hear more if others have deployed this approach for more critical loads or more demanding users.

Other uses

There are loads of potential uses for desktop virtualisation like this. One of the most obvious ways that it could be used would be to allow a team member to access an isolated development environment (or one of its applications) over the LAN. This wasn’t possible with my earlier design, and frankly, you don’t want to do a lot of this, but sometimes it can be really useful.

I realise this is conceptually difficult, so feel free to ask questions if this hasn’t been my clearest post. I’ve skipped some steps that are required to deploy the Remote Desktop Services. That’s another topic that is beyond the scope of this post. Again, I’m happy to answer questions about that, although I’d recommend hitting TechNet for a few hours before diving in. There’s loads of new functionality here and you may find another option suits your needs better. To the initiate, the Microsoft App-V, MED-V, MDOP story might look a bit “nice to have”, or possibly too complex, but this really isn’t complicated to implement and engaging with this new technology has opened my mind to other design options that I had previously shut out. Definitely worth a look.

When I initially created our SharePoint 2010 beta development environments, I built them in a Workgroup, for many of the reasons that I chose to do so in 2007 (see the Workgroup Development section in Part II of my series on SharePoint development environments). Now, the SharePoint 2010 Configuration Wizard (psconfigui) throws an error when choosing the Complete install option in a Workgroup. I was able to get past this error by following the suggestions on the Microsoft From the Field blog, but a few weeks down the road we’ve pinned down two issues (as confirmed by Neil, the author of that post).

1. Search will crawl successfully in this configuration, but the query role will never initiate. Queries from web applications that consume this Service Application will produce an error, which you will be able to trace to an application event log 6398 error and the key event in ULS, “The SDDL string contains an invalid sid or a sid that cannot be translated.” As Victor Magidson from Microsoft puts it on this Technet thread, this error occurs when a, “topology activation job is trying to create a propagation file share.” In short, the query role never finishes initialising so it will never serve queries. This only occurs on the single server Complete installation and deleting/re-creating the Service Application yields the same result.
2. The User Profile service application will also be useless*, but it always would have been since you couldn’t import users from the local user database in 2007 either. However, this has wider implications in SharePoint 2010 because of the social computing features, which developers probably won’t be able to live without.

All this means that I need to consider rebuilding our development environments in a domain or we need to use the Simple installation. In the past I always avoided the Simple installation because it uses system accounts like Network Service for application pool identities, so it took some re-acquaintance to identify some of its limitations. It has no configuration options (thus the use of local system accounts), the User Profile Service Application does not start (presumably per Forefront Identity Manager 2010 installation issues as seen in other topologies) and it uses SQL Server Express (which developers aren’t very fond of), so we’re unlikely to pursue this option. Accordingly, we’re reviewing the ways that we can re-create the development environment in a domain.

Why don’t I just make the development machine a domain controller? This isn’t clear-cut and there are benefits to the simplicity of it, but ultimately I think it’s a bad idea because:

• Domain Controller security is bad for development. It means developers will be coding as Domain Admins and they will be doing so on a machine with Domain Controller security policies. This is just a mess.
• SQL doesn’t like to run on a DC.
• Running a DC, SQL and SharePoint on the same machine creates a massive load of service start-up contention and sometimes the environment will start from an unstable point because dependent services will not be ready when a depending service tries to start.
  • This also increases start-up time considerably, which is a big concern when using Hyper-V on a laptop, where we don’t have Sleep/Hibernate.
• Adding Visual Studio to this mix causes known performance issues. The machine simply can’t keep up with doing all of this.

Why won’t we use a central development domain? Because then we can’t clone the VMs. We don’t just clone standard builds, we also use the same VM for each developer/consultant/architect on a single project. We may opt for this approach eventually, but I think the investment in automation is pretty considerable for it to be efficient with our team development requirements.

So I’m probably going to separate the DC on to a different server for now. I’m unlikely to add a third server for SQL, as even though it might be faster, I think the complexity of managing networking and snapshots across three servers is undesirable for developers, if we can get by with two.

All of this underlines the reasons why I didn’t publish any SharePoint 2010 build guidance yet. Some things don’t become clear until you get your hands dirty. I’ll revisit this topic as soon as I feel comfortable that we’re pinning things down, but for now we’re still adapting our methods.

*I’ll note that I haven’t properly tested Claims Based Authentication against a single server complete installation, so it’s conceivable that you could import non-A/D users in to this configuration, but you’d still need to live without Search.

A couple of weeks ago I was working simultaneously on my Windows Server 2008 R2 laptop with Hyper-V (the same laptop build that’s been previously mentioned) and a Windows 7 x64 build that I was using for testing, when I noticed severe but intermittent network problems on both machines. After a fair amount of head scratching, I noticed that the two laptops had duplicated MAC addresses. Blatantly that shouldn’t happen, as the whole point of a MAC address is to provide uniqueness. The most perplexing issue was that the addresses conflicted across two different operating systems. However, it happened. Both wired adapters on the two machines had the MAC address 00-21-9B-DC-8E-0B. I uninstalled the wired adapter on the Windows 7 machine and scanned for new hardware. When the device reinstalled the problem went away.

Since then I’ve been poking around a bit and I’ve found this old TechNet article from December 2007 in which Wes Miller says:

In addition to changing the SID and the machine name, you also need to change certain values that may be specific to the virtual computing technology you’re using. In particular, you need to change the MAC address (the unique ID for networking devices). Plus, many virtual applications also have their own unique identifier. Most store these in their own machine configuration files, so you’ll want to know how to manipulate those entries (and maintain their validity). Note that many virtualization products that support Pre-Boot Execution Environment (PXE) key the SMBIOS UUID based on their own unique ID—emphasizing the need to change this (or let the virtualization software change it for you, if supported) if you’re joining it to a domain; otherwise, managing WDS or RIS-client systems can become impossible (if GUIDs conflict). Most of the virtualization solutions I’ve worked with can have severe networking problems in the case of duplicate MAC addresses; so if you are not just moving a virtual machine, it’s very important that you change the MAC address if the virtualization software does not do it for you.

That’s by no means conclusive, but I’m not turning up much else. So if you’re encountering network problems with WDS images that began their life as a virtual machine, you may want to consider automating the un/re-installation of the NICs post-deployment or addressing that with WAIK somehow. Otherwise just make sure to do it manually before giving the machine to the recipient, or people will wind up with a rather frustrating network issue.

If we get around to doing any more testing of this or gather more evidence from internal deployments, I’ll post the findings here. In the mean time I thought I’d update the ongoing saga here.

Issues

Guest user accounts

Guest virtual machines have been configured in a workgroup in order to conserve resources that would be spent on domain services. Additionally, developing on a domain controller is less than ideal for a number of reasons including performance tuning, administrative complexity, start-up times and security.

I created the development virtual machine with 160 local user accounts that have been logged on to the portal and the MySite application in order to create a basic profile. If there is a need to script creation of local user accounts, the net user command will be useful. However, this will be of limited assistance for complex profile requirements, since there is no way to synchronise with a directory and since the local users have no associated profile data, but it may be helpful for testing or demonstration.

If LDAP user accounts or other directory objects are required for development purposes (user profiles for instance), consider using Active Directory Lightweight Directory Services. This is the successor to Active Directory Application Mode (ADAM) in Windows Server 2003. It is a Windows LDAP directory that supports user and group objects without a full-blown domain infrastructure.

There will be some scenarios when a full domain services infrastructure is required for development. In those cases it may be preferable to run a second virtual machine as a domain controller.

Hibernate and Sleep

Hibernate and Sleep are disabled automatically when Hyper-V is installed. This is by design. Hyper-V disables this functionality, as the guest virtual machines could be damaged by a Hibernate or Sleep operation in the host if they were not saved gracefully. If, on the other hand, all virtual machines had to be put in to a saved state before a host machine could be put to sleep or hibernated, this would mean extending the wait time for these operations to unacceptable levels, as they are also automatically triggered by low battery warnings. Unfortunately we need to live with this behaviour.

Do not travel with a running laptop

Putting a running laptop in a bag will cause it to overheat quickly and is likely to damage hardware.

Improvements to Start-up and Shutdown times

These builds should start up and shut down in less than two minutes (closer to 90 seconds). Keep in mind that virtual machines can be safely saved and work can be resumed quickly when the machine is restarted. Since all of the development work will be taking place inside the virtual machine, this should reduce the Hibernate/Sleep annoyance.

Virtual PC won’t run on Windows Server 2008 R2

Windows Virtual PC will not work on Windows Server 2008 R2, as it was designed specifically for Windows 7. Earlier versions of Virtual PC may install on Windows Server 2008 R2, but they will not co-exist with Hyper-V, so do not install them.

Hyper-V role won’t work after SysPrep

This shouldn’t be an issue, as we have set up automated deployment, but it’s worth noting that this is a known issue. There are time-consuming work-arounds to fix some of the problems that this will cause, but they are best considered as a last resort.

Colour management

Colours are limited to 16-bit in Hyper-V guests. If a fuller spectrum is required, it should be possible to test in full colour in a browser on the host.

Resolving host names from an internal domain

During our pilot we identified that fully-qualified domain names resolved successfully but host names would not resolve without the full domain name. To satisfy this requirement we have added our internal DNS suffixes to the ICS Connection inside the development virtual machine.

Manually adding DNS suffixes

If a network adapter in a guest virtual machine loses these settings by deletion/re-creation of the adapter, or for some other reason, the setting can be re-entered as follows:

• Go to the IPv4 properties on the ICS Connection and select Advanced.
• On the DNS tab select the Append these DNS suffixes (in order) radio button.
  • Add internal.domainname.local and other.domainname.com.
• Un-tick the DNS registration box.
• Select OK, OK and Close.
• Make sure that this change is captured in all snapshots as necessary.

Internet Explorer (64-bit version)

Adobe flash player does not currently support 64-bit browsers. You’ll have to use the 32-bit IE or another browser if you want to view flash files. We recommend using the 32-bit version by default.

Hyper-V Manager UAC prompt work-around

If the UAC prompt on Hyper-V Manager is annoying, try launching Server Manager and navigating to Hyper-V in the Roles node. This has the added benefit of exposing the Hyper-V event log messages and service states in that top Hyper-V node. These are not visible in Hyper-V Manager. Awareness of these messages and the service statuses will help to resolve Hyper-V issues faster.

Test DVD burning

In our pilot we identified that the DVD burner drivers don’t work for burning in Windows Server 2008 R2 on a Dell XPS M1330. This was also true on a Lenovo laptop. Chipset updates, driver updates and a Microsoft KB registry hack all failed to make a difference. The Matshita (Panasonic subsidiary) site does not support the products directly (they point to the laptop manufacturer). Dell and Lenovo had not released new drivers when we launched. As DVD burning has changed in Windows Server 2008 R2 this may have a wider impact.

Bluetooth doesn’t work

The Bluetooth stack is missing from Windows Server 2008 and Windows Server 2008 R2. In Windows Server 2008 there were fairly elaborate means of porting the stack from Vista, but results appear to be spotty at best.

WorkItemTypeDeniedOrNotExistException when trying to open work items

This error occurred in the first release of our guest build because I installed Visual Studio 2008 SP1 before the Team Foundation Client (TFC), so the TFC did not get upgraded. The fix is to un/re-install Visual Studio 2008 SP1, or to make sure that the TFC is installed before Visual Studio 2008 SP1.

NUMA nodes and RAM allocation

It is important to not exceed NUMA node limits when assigning RAM to virtual machines, although this will not apply to many laptops, as most will have an SMP architecture. It is beyond the scope of this post to go in to NUMA nodes in great detail (and in truth, my understanding of it does not reach beyond a few hours of research), but the limits in your environment should be understood so that performance does not suffer. As a starting point it’s worth confirming the type of CPU architecture and looking at this in more detail if it is NUMA. The performance and capacity requirements for Hyper-V document on TechNet explains this well:

Configure the correct amount of memory for Hyper-V guests. During the testing, no change had a greater impact on performance than modifying the amount of RAM allocated to an individual Hyper-V image. Because memory configuration is hardware-specific, you need to test and optimize memory configuration for the hardware you use for Hyper-V.

The initial goal of the testing was to make the Hyper-V image as similar as possible to the physical hardware image against which it was being compared. Based on that goal, the Hyper-V images were originally allocated 32 gigabytes (GB) of RAM, which was the same amount of RAM as was on the physical servers being tested. However, the initial test results showed that with that configuration, the Hyper-V images could sustain a load that was only about 70 percent of the load on the physical hardware. After investigating the Event Viewer on the Hyper-V host machine in the Windows Server 2008  Custom Views, Server Roles, Hyper-V Events, it was discovered that the RAM for the Hyper-V images was being spread across multiple non-uniform memory access NUMA nodes. This information confirmed that performance declined when memory was allocated across nodes. After trying different configurations it was determined that for the hardware being used, 8 GB of RAM was the maximum that could be allocated to a Hyper-V image without crossing NUMA nodes.

To reiterate, this means that in Microsoft’s tests, Hyper-V performed significantly worse with 32GB allocated to a virtual machine than it did with an 8GB allocation. The exact size of the NUMA node boundary will vary by vendor, so make certain to gain an understanding of the number of nodes in your system. Divide the total RAM by the number of nodes in order to find the memory limits of a virtual machine. This does not mean that additional virtual machines can’t be run beyond a NUMA node boundary, if there is sufficient RAM available. The node boundary is the limit of optimal process performance. Beyond this limit, the virtual machine will suffer from degraded performance because it needs to use memory from an alocal address space.

However, NUMA isn’t the only thing to worry about when finding an optimal RAM allocation. Based on test results during our pilot we could push our virtual machines to up to 2250MB RAM, depending on the amount of activity in the host. In some cases it may be possible to get up to 2500MB RAM for a virtual machine on a 4GB RAM system, but this was not consistently achievable in our tests. If it’s necessary to achieve that, the virtual machines should be started up soon after booting and before any major client application activity is started on the host machine. Client application activity should be kept to a minimum when allocating this much RAM to virtual machines. We also found that host performance was often reduced to an intolerable level whenever there was less than 2GB RAM available to the host for an extended period of time. 1.75GB RAM may be achievable, but this should be tested extensively for your needs.

Additionally, saving a virtual machine’s state becomes risky when there is less than 2GB available to the host, as the machine will not resume from the saved state if there is insufficient resource available to it.

Periodic but routine loss of connectivity on the host machine

As I’ve been tracking here, we’ve documented repeat problems with periodic (but routine) loss of connectivity on the host machine. This is still an open issue. More info here:

Routine loss of connectivity on a Hyper-V host’s external connection
More on routine loss of external network connectivity on Hyper-V hosts (not guests)

Hyper-V performance suffers during graphics-intensive operations

This has been covered by Ben Armstrong in considerable detail and I’m continuing to track it:
Hyper-V graphics performance and SharePoint 2010 development
Hyper-V graphics performance is on the way… if you need a new laptop
The definitive word on Hyper-V high-end graphics performance

Aero Glass

Unfortunately, due to the graphics performance issues in Hyper-V mentioned above, there is a significant graphics performance hit when using Aero Glass. This does not slow down overall systems performance, but graphics-heavy operations will suffer in most Hyper-V environments. To this end, we do not recommend installing Aero Glass, but if you want to put it to the test feel free.

How to enable Aero Glass

• Make sure the Desktop Experience is activated
  • On the Dell XPS M1330, make sure BIOS A14 or later is installed
    • Confirm the latest NVIDIA drivers for Windows 7 x64 are installed
    • Turn on the Desktop Window Manager Session Manager service and switch to automatic start
    • Turn on the Themes service and switch to automatic start
    • Switch to an Aero theme

Sidebar

It is also possible to add the Windows 7 Sidebar to the host dekstop. We haven’t tested this extensively enough to provide documentation on the best approach, but we have done it successfully. If there is sufficient interest in this technique I will add a follow-up post in future.

Storage

Be prepared for snapshots to increase the local storage requirements considerably. Some of our developers are legitimately struggling to work on three projects concurrently with 300GB local storage. One option we are considering is eSATA over PCMCIA as a means of increasing total spindle speed and storage but we have yet to begin testing this approach. We are specifically interested in eSATA since Hyper-V does not support system VHDs on USB. If we pursue this option I’ll post the results.

Results

Putting things in perspective

It’s worth repeating that we’re asking this system to do many things it was not intended to do. It is a server operating system with an enterprise virtualisation technology. The Microsoft virtualisation team will tell you that Hyper-V was not designed with developers in mind. To quote Ben Armstrong:

As is being discussed at length here – Hyper-V does not play well with high-end video cards (which are far more common on desktops than servers).  Hyper-V also disables sleep and hibernate, as well as increasing the power utilization of the computer.  All of these things would need to be addressed before we could even consider putting Hyper-V in a desktop product.

In short, we bent this system to development needs because of the strength of the technology, despite these imperfections. There are fundamental compromises that can’t be avoided when using a server operating system as a mobile workstation but we believe that we can deliver SharePoint projects as a team better with this technology than without it.

Hardware

Whether laptops are the ultimate hardware solution is a different can of worms, which I’ve chosen to avoid in this series of posts. I’ve tailored the approach to laptops since that is what we have and the approach can be ported to workstations or shared virtual infrastructure.

The developer experience and the bottom line

There’s no question that using snapshots, import and export in Hyper-V adds a complex tier to the development experience and there will be a learning curve for those who are less familiar with virtualisation or don’t use the advanced features often. However, we have achieved an immediate and measurable gain in stability and environment consistency through the use of standard builds, snapshots and exported project-defined environments.

Conversely, it’s worth keeping in mind that as desirable as standardisation is, there are times when it hinders more than it helps and on those occasions a non-standard build may be more appropriate. Considering alternative builds is a much less cumbersome proposition with  the combination of WDS, Shrink Volume and Dual-booted systems or the new Native boot from VHD. The key consideration to keep in mind is that most other approaches will entail a sacrifice of what I lump together as the “management benefits” of Hyper-V (snapshot, import and export). For instance, you may consider allowing a team to develop on native operating systems for a project, but then a team member may lose a day if they need to rebuild their system, or the support team may need two days to build an environment in Hyper-V later on, or a team member may need to split her time with a team who use Hyper-V for their project, or the original project may fork. Memories of project difficulties gone-by come flooding back. While it’s always worth considering options, if you spend time identifying a standard approach for your business, it’s probably best to stick with it unless there’s a truly compelling reason not to.

Where’s the SharePoint 2010 build?

In short, we’re working on it. I’ve produced a new SharePoint 2010 beta virtual machine for this environment but we’re not yet ready to publish build guidance. Stay tuned. Additionally…

SharePoint 2010 memory requirements

We have validated this environment for use with SharePoint 2010 virtual machines, but performance will be inadequate for development unless 4GB RAM can be dedicated to the SharePoint 2010 VM. It will run with 2GB but it will be too slow to do anything beyond lightweight demonstration. Also, be sure to check if your system has a NUMA CPU architecture. If it does, only allocate memory within NUMA node boundaries, as performance degrades when accessing memory from alocal nodes. See the performance and capacity requirements for virtualising SharePoint guidance for an introduction to the issue, but be aware  that NUMA architecture is less common in laptops so it may not be an issue.

Virtual networking recap

As explained in part two of this series, we have created  three virtual networks on the Hyper-V host, although this development virtual machine will only have two network adapters. The first will be “plugged in” to the Hyper-V ICS Network. The second is “plugged in” to the Hyper-V Internal Network. We plug the Hyper-V External Network in for testing purposes only. For more information about how I’ve designed these networks please refer to the design post.

Working with virtual networks

The guest virtual machine’s “ICS Network connection” will receive the host’s LAN or VPN connectivity (and an IP address via DHCP) from the host’s adapter on the Hyper-V ICS Network. The second adapter is “always on”, in that communication between the host and the guest will be completely separated from the host’s provision of ICS to the guest. This second adapter has a fixed IP address on a network that’s dedicated to providing communication between guest VMs and a resilient, permanent connection in the host. This enables:

• Persistent network sharing.
• RDP connections to guest VMs.
• HOSTS file entries and BackConnectionHostNames security.
• Multiple IP addresses on static adapters in guests, for multiple sites with SSL, etc.

Static IP address allocation on the ICS connection in the guest is not a supported configuration and results in instability.

System Build

Installation

Create a new virtual machine in the location of your choice. After the virtual machine has been created, modify these settings:

• Assign multiple processors to the virtual machine (for most laptops you will want to assign all of them).
• Assign sufficient RAM, leaving at least 2GB RAM for the host machine (1.75GB bare minimum – host performance is too slow with only 1.5GB available).
• Connect the Hyper-V ICS Internal Network to the first NIC.
• Connect the Hyper-V Internal Network to the second NIC.
• Point the DVD drive at an ISO file or installation media for Windows Server 2008.
  • It’s also possible to network-boot, following a similar approach to the host deployment from WDS. To network-boot a Hyper-V machine, add a legacy network adapter (it must be legacy) and modify the BIOS settings to boot from the network.
  • We could have installed Windows Server 2008 R2 as we did on the host, but at the time of our pilot it was still only in Release Candidate, so we opted to stick with a tested OS in order to compare the development experience with our current environments. I built our SharePoint 2010 Technical Preview and Beta environments on Windows Server 2008 R2 RTM without problem.

Turn on the virtual machine and install Windows Server 2008 Standard. On completion, log in for the first time, insert Hyper-V Integration Services from the Virtual Machine Connection’s Action menu and reboot when prompted.

Initial configuration

These steps establish basic connectivity and improve usability:

• Log in to the new virtual machine and enable remote desktop connections using any version (in order to support older versions of Royal TS). If all versions of Remote Desktop are current then select the secure option.
• Test that the ICS connection is able to retrieve an IP address and that the connection is receiving connectivity from the host machine. If there are any problems connecting to the LAN or web, make sure that the connection is shared in the host and that the ICS connection in the guest is on DHCP. Also check the firewall settings. No settings should need to be modified but make updates if necessary.
  • Temporarily plug the Hyper-V External network in to this adapter for testing if needed. This is a good way of identifying if a problem relates to ICS itself or if it’s a guest firewall issue.
• Add DNS suffixes to the ICS connection if you want to be able to resolve host names on your LAN (without having to use the fully-qualified domain name). ICS will not pass through primary or connection specific DNS suffixes.
  • Go to the ICS Connection’s IPv4 properties.
  • Select Advanced.
  • On the DNS tab select the Append these DNS suffixes (in order) radio button.
  • Add domain suffixes for any named resources that should be resolved using host headers.
• Configure IP settings on the Internal Connection in the guest.
  • Configure IP addresses as 192.168.200.100.
  • Set the Subnet Mask as 255.255.255.0.
  • A default gateway and DNS settings are not necessary.
• Remove IPv6 from all connections in the host and guest. See earlier guidance for more information.
• Give all adapters sensible names.
  • I chose “ICS Connection” and “Internal Connection”.
• Test browsing to \\192.168.200.1 to test connectivity over the Internal network.
• Optionally, turn off IE Enhanced Security Configuration (ESC) for administrators.

Patching

• Apply Windows Server 2008 SP2.
  • Reboot.
• Patch current.
  • Reboot if prompted and patch again if patches are available. Reboot again if needed and repeat until current.

Shutdown and snapshot

Shut down the virtual machine and take a snapshot. Rename the snapshot with an appropriate name, such as “Windows installation and patching complete”. You will be able to roll back to this state if you have problems with the Windows configuration.

Adding Prerequisites

Adding Web Server (IIS) role

Start up the virtual machine and add these Windows Role services:

• Web Server
  • Common HTTP Features
    • Static Content
    • Default Document
    • Directory Browsing
    • HTTP Errors
    • HTTP Redirection
  • Application Development
    • ASP.NET
    • .NET Extensibility
    • ISAPI Extensions
    • ISAPI Filters
  • Health and Diagnostics
    • Logging Tools
  • Security
    • Basic Authentication
    • Windows Authentication
    • Request Filtering
  • Performance
    • Static Content Compression
    • Dynamic Content Compression
• Management Tools
  • IIS Management Console
  • IIS Management Scripts and Tools
  • Management Service
• Windows Process Activation Service
  • Process Model
  • .NET Environment
  • Configuration APIs

Adding Features

Add these Windows Features:

• Desktop Experience
• Optional: Windows System Resource Manager (and Windows Internal Database)

Installation of the Desktop Experience will prompt for reboot. It is not necessary to do so immediately. Note that when the reboot happens, Windows will restart a few times. This is a normal part of the Feature configuration.

Additional preparation

HOSTS file entries

Created the following HOSTS file entries for the web applications:

192.168.200.100                SSPAdmin

192.168.200.100                MySite

192.168.200.100                Portal

DisableLoopbackCheck or BackConnectionHostNames updates

Either disable the Loopback Check or add BackConnectionHostNames entries. Since ICS prevents inbound connectivity to these virtual machines we feel comfortable disabling the loopback check, as this would be a considerable annoyance to developers otherwise. For unisolated systems I would always recommend putting the BackConnectionHostNames entries in place. Spencer Harbar has summarised the background information best.

To disable the Loopback Check, add a new DWORD DisableLoopbackCheck value “1” to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa

Creating the Source repository in the host and sharing to guests

To understand these steps it is worth reviewing the Local source code storage on the host machine section of the design post.

• Create a new user account in the guest machine called SourceShare, with same password as the account that was created by the WAIK unattend file in part four of this series. This password should be strong, as the share will be visible to authenticated users on the host machine.
• Confirm that the CreateSourceShare.bat script from part four has created a folder on the host system that is shared it to the SourceShare user with full control. Remove “Everyone” from the sharing permissions and replace with “Authenticated Users”.
• In the guest virtual machine, map the Z drive to \\192.168.200.1\Source, specifying the SourceShare account credentials and that it should be reconnected at login. Save the password.
• Test the connection by opening the Z drive. If there are problems opening the drive, browse to \\192.168.200.1\ and troubleshoot from there.
• If users have trouble with the strength of the password, allow them to change it to something that they will remember, but make certain that it is changed in both the host and the guest and make sure they know that the change will need to be applied to all snapshot states and any other virtual machines that use this repository in the host.

Note: if you would prefer to store source code in the virtual machine it is not necessary to follow these steps. The benefits of preserving code outside of the snapshot state need to be weighed up against the drawbacks of rolling back to a state when some of that code did not exist, and users of the system need to be aware of the implications of using either approach. Clearly this approach adds some complexity and any approach should be validated for your needs.

Create local user accounts for services and application pool identities

• SQL service accounts (which should be for obvious purposes):
  • SQLSvc
  • SQLAgent
  • SQLAnalysis
  • SQLReporting
• SharePoint service accounts and application pool identities:
  • MossSetup (installation account)
  • CentralAdmin (farm account)
  • SSPAdmin (Shared Services Administration site application pool identity)
  • MySite (MySite application pool identity)
  • Portal (Portal application pool identity)
  • SearchService (MOSS Search service account)
  • SharedServices (the SSP web service identity)
  • ContentAccess (the crawl account)

Assign the MossSetup account local administrative rights.

Activate windows

Add a valid Windows license and activate the system. Aside from obvious compliance issues, if this is not done you will see repeated 12321 Security-Licensing-SLC “Token-based Activation failed” warnings in the event logs.

Shutdown and snapshot

Shut down the virtual machine and take a snapshot. Rename the snapshot with an appropriate name, such as “Windows configuration complete”. You will be able to roll back to this state if you have problems with the SQL installation.

SQL 2008 Setup

Installation

Start up the virtual machine and run the SQL 2008 installer. Select all features.

• Give the instance a name or choose the default instance name according to preference.
• Set up SQL services under these accounts:
  • SQLSvc
  • SQLAgent
  • SQLAnalysis
  • SQLReporting
  • Everything else can run as Network Service.
• Disable Analysis, Reporting and Integration services. We do not run these services by default in order to optimise performance. However, we do install the services to speed up deployment as needed. Use as necessary in your environment.
• Select Windows Authentication.
• Select the local Administrator account as a Server Administrator.
• Install Reporting Services in SharePoint integrated mode.

After installation completes, launch the SQL Management Studio and assign the MossSetup account DBCreator and SecAdmin rights. It will not be necessary to assign other permissions directly in SQL, as the setup account will assign DBCreator/SecAdmin rights to the farm account.

Patching

Add the SQL Server 2008 Feature Pack and SQL Server 2008 SP1 and. Reboot as necessary and patch with the latest Cumulative Update.

Shutdown and snapshot

Shut down the virtual machine and take a snapshot. Rename the snapshot with an appropriate name, such as “SQL 2008 configuration complete”. You will be able to roll back to this state if you have problems with the MOSS installation.

MOSS deployment

Installation and initial configuration

Start up the virtual machine. Log in with the MossSetup account and follow these steps.

• Run the MOSS installer with (at least) SP1 slipstreamed. Select the Complete Server Type.
• When the installer completes, un-tick the “Launch SharePoint Products and Technologies” wizard tick box.
• Open a command window, running as Administrator and navigate to the BIN in the 12 Hive. Run PSCONFIG –cmd configdb –create to manually specify friendly names for the Central Admin databases and to assign the CentralAdmin farm account as the user. Specify both the Central Admin configuration and Content databases or they will receive an awful GUID for a name. Find more information about PSCONFIG on TechNet.
• Launch the SharePoint Products and Technologies Configuration wizard from the start menu. Retain the connection to the newly-created farm databases and proceed through the configuration wizard according to your build requirements. I recommend that you specify a standard Central Administration port if you use one and authenticate using NTLM.
• Once the wizard completes, launch Central Administration to confirm that the site is up and running.
• There are tips and tricks that you may want to include for your developers, for instance Andrew Connell’s suggestions and an interesting use of junction points from the Share Notes blog.
  • Consider putting a desktop, task bar or start menu shortcut to the 12 hive.
  • Add an environment variable for the 12 hive’s BIN.
    • Go to the machine’s Advanced System Properties (right-click My Computer and select Properties).
    • Click the Environment Variables button.
    • In the System variables pane, scroll down to Path and select “Edit”.
    • Clicking in the Variable Value will put the cursor at the end of the line. Type a semi-colon.
    • Copy and paste the path to the 12 hive’s BIN, typically:
    • C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\

Shutdown and snapshot

Shut down the virtual machine and take a snapshot. Rename the snapshot with an appropriate name, such as “MOSS installation complete”. You will be able to roll back to this state if you have problems with the farm configuration.

Farm build and configuration

Start up the virtual machine and launch Central Administration from the Start menu or navigate to it in Internet Explorer.

• Enable Intranet Settings for the Central Admin site. This option should appear as a prompt directly beneath the tabs.
• Grant Launch and Activation permission to the local WSS_WPG and WSS_ADMIN_WPG groups in the IIS WAMREG DCOM component. This prevents system error 10016. See KB920783 for more information.
  • If necessary, do the same thing on the oSearch DCOM componet, per KB953137.
• Enable the MOSS Search service, running under the <MACHINE NAME>\SearchService account with “Reduced” performance, using all WFE servers.
  • If the account name is entered without the <MACHINE NAME> prefix, you will likely see this error:
    “An unhandled exception occurred in the user interface.Exception Information: OSearch (SearchService)”.
• We have not configured any other central administration settings in order to provide the farm in the cleanest state possible. That said, perform any additional configuration here as needed, but be aware that many settings are not suitable for all projects.

Create the Shared Services Provider

Configure the farm’s shared services as prompted by the left navigation. This will step through the creation of the SSP administration and MySite web applications and then the Shared Services web service itself.

• Create a new SSP administration web application when prompted.
  • Use the host name and load balanced URL that was set up above: http://SSPAdmin
  • Use the application pool identities as created above: SSPAdmin
  • Select NTLM authentication, unless there is a reason not to.
  • Give the content database for the web application an appropriate name, such as WSS_CONTENT_SSPAdmin, according to internal naming conventions.
• Create a new MySites web application when prompted.
  • Use the host name and load balanced URL that was set up above: http://MySite
  • Use the application pool identities as created above: MySite
  • Select NTLM authentication, unless there is a reason not to.
  • Give the content database for the web application an appropriate name, such as WSS_CONTENT_MySite, according to internal naming conventions.
• Proceed with the SSP configuration:
  • Use NTLM authentication.
  • Specify the SharedServices application pool identity.
  • Do not use SSL, as this traffic is not leaving the server.
  • Typically in a stand-alone devlopment environment it should be OK to use the default SSP database naming suggestions, but adhere to internal standards where they apply.
• Wait for the configuration wizard to complete. This can take some time.
• Visit the SSP administration site to confirm that the web application has been successfully created.
• In Central Administration, navigate to Self-Service Site Management and enabled Self-site creation for the MySite web application.
• Visit the MySite link in Central Administration to confirm that the web application has been successfully created.
• Log in to the SSP Administration site and take these steps:
  • Specify a default content access account: ContentAccess
  • Grant full personalisation services rights to the local administration account.
    • This account can be used to grant rights to yet-to-be-created local users of the system once they receive this build, if desired.

Create the web portal

In Central Administration create a new web application.

• Use the host name and load balanced URL that was set up above: http://Portal
• Use the application pool identities as created above: Portal
• Select NTLM authentication, unless there is a reason not to.
• Give the content database for the web application an appropriate name, such as WSS_CONTENT_Portal, according to internal naming conventions.
• Create a root site collection using the collaboration portal template. Make the local administrator account and the Moss Setup accounts site collection administrators – to be changed by local users on receipt of the build.
• Visit http://portal to confirm that the site collection has been successfully created.

Shutdown and snapshot

Shut down the virtual machine and take a snapshot. Rename the snapshot with an appropriate name, such as “MOSS configuration complete”. You will be able to roll back to this state if you have problems with the patching.

Patch current

Start up the virtual machine and patch the environment to whatever state is needed. We typically recommend patching current. During the course of our pilot we used the April 2009 Cumulative Update. When we released the base virtual machine to our entire development team we launched with the June 2009 Cumulative Update. Once patched current, test that all sites are still working as expected.

Full patching guidance is available on the Update Center for Microsoft Office, Office Servers, and Related Products site.

Shutdown and snapshot

Shut down the virtual machine and take a snapshot. Rename the snapshot with an appropriate name, such as “June 2009 CU applied”. You will be able to roll back to this state if you have problems with developer tool installation.

Tool installation and export

Developer Tool installation

Start up the virtual machine and install the common development tools for your organisation. These are the standard tools used by our developers. There are also some licensed products that are installed for some users outside of the scope of the base build.

• Microsoft Office SharePoint Designer 2007 SP2
• Microsoft Visual Studio Team System 2008 Developer Edition (we created a separate snapshot for users that need Visual Studio 2005 rather than running them concurrently)
• Microsoft Visual Studio Team System 2008 Team Explorer
• Microsoft Visual Studio Team System 2008 Team Foundation Server MSSCCI Provider
• TFS Power Tools
• Microsoft Visual Studio Team System 2008 Developer Edition SP1
• Microsoft Visual SourceSafe
• WSPBuilder Extensions 1.0.5
• StyleCop
• Fiddler2
• Sandcastle
• Sandcastle Help File Builder
• HTML Tidy
• PDF X-Change PDF Viewer
• Firefox with Firebug
• Safari
• Chrome
• Opera
• SharePoint Manager 2007
• Microsoft Silverlight
• Adobe Flash for IE
• Adobe Flash for Firefox, Safari and Opera
• Microsoft Forefront Client
• Test Mail Server

On completion of installation, defragment the disk and wait for completion.

Shutdown and snapshot

Shut down the virtual machine and take a snapshot. Rename the snapshot with an appropriate name, such as “Developer tools installed”. You will be able to roll back to this state if you have problems with the farm configuration.

Snapshot branches

If you will be using different versions of Visual Studio (or anything else) it’s worth creating distinct branches for each version. You could adopt an approach like this:

• Install all of the independent tools, shut down and take a snapshot, renamed to something like “Developer tools except VS installed”.
• Install Visual Studio 2008 and patches. Defragment, shut down and take a snapshot, renaming to something like “Visual Studio 2008 installed and patched current”.
• Apply the “Developer tools except VS installed” snapshot. Install Visual Studio 2005 and patches. Defragment, shut down and take a snapshot, renaming to something like “Visual Studio 2005 installed and patched current”.
• The two Visual Studio snapshot are now siblings – children of the “Developer tools except VS installed” snapshot node. A third sibling could be created in this same manner for Visual Studio 2010 Beta 2 (for instance).

We install Visual Studio last in the installation sequence in order to maintain consistency among these variants as deep in to the snapshot tree as possible. To give another example, I’ve just built a SharePoint 2010 Beta environment with Visual Studio 2010 Beta 2. In that environment there was only one instance of Visual Studio but two SharePoint branches – one for SharePoint Foundation and another for SharePoint Server 2010. In that tree I installed all developer tools and Visual Studio before installing either SharePoint edition. If you have similar branching needs, plan to install those items last and structure your snapshot trees accordingly.

Export

You may choose to export the entire virtual machine rather than the final snapshot, depending on how standard you want the final build to be. If a developer can role back to an earlier state then the standardisation of the build is at risk. In our case, we’ve exported the final snapshot (or each of the final sibling nodes in the snapshot tree, since some users use Visual Studio 2005). This state is vanilla enough that it can be used as a common starting point for a wide range of projects. It has even been used as a starting point for projects that extend beyond SharePoint to accessibility solutions, Commerce Server and BizTalk.

This is not to say that this build fits all sizes, but it does fit most of them while obtaining the benefits outlined in the first post in this series. Some of the limitations of this approach are discussed in the next part of this series.

Last week we built a Windows Server 2008 R2 Hyper-V host that we used for our SharePoint 2010 launch event at Microsoft London. We were practicing the demonstration from connected Windows 7 laptops but we also wanted to understand the limitations of presenting from the host itself, should something go wrong with the networking. For the most part this worked fine, but we ran in to problems when we tried to run the media web part. The dialogue box would pop up as normal, but there was no option to Play. If I clicked View I would be prompted to download the .WMV file. At this point I realised that this hardened server did not have the Desktop Experience enabled, so it didn’t have a default media player.

After installing the Desktop Experience and rebooting (note: this does a few reboots), the pop-up looked the same; the Play button was still missing. Now if I clicked View it would launch in Windows Media Player.

Eventually it occurred to me that the server might be missing Silverlight. We install it by default, so it took some time for me to catch on. I checked the installed updates and sure enough, it was missing. When I tried to run Windows Update it revealed that Silverlight was the only availably update. I tried to install it and I quickly got error 80244019. Searching for a solution suggested a few possible answers – mostly network/DNS-related. To expedite things I visited the Silverlight site and installed it manually. I believe the problem may have been DNS or proxy-related but since all other Windows updates installed fine it’s a bit of a mystery.

Installing Silverlight manually did the trick. The Play option returned in the pop-up and video played normally. Out of curiosity I removed the Desktop Experience and tested again. Everything continued to work normally. All of this is totally clear in retrospect, but with the new technology cocktail that is SharePoint 2010, Silverlight 3, IE8 and Windows Server 2008 R2, it’s easy enough to loose sight of the obvious.

• It’s not an issue on processors with SLAT, but these are only just hitting the market in laptops in the near future
• It’s not an issue with the SVGA driver
  • I’ve asked if the SVGA driver might ever offer multi-monitor support. He’s looking in to it. This might be a great compromise until processors with SLAT become ubiquitous
• This same problem occurs in all native Hypervisors
  • Virtual PC and VMWare Workstation do not have the same problem but they are Type 2 hypervisors and do not offer the same performance as Hyper-V

So… there’s still no conclusive solution but it’s good to have the full context of the problem. For more background on why this matters for SharePoint see my previous post on the matter. ]]> http://tristanwatkins.com/index.php/the-definitive-word-on-hyper-v-high-end-graphics-performance/feed/ 1 http://tristanwatkins.com/index.php/building-a-sharepoint-20072010-development-environment-part-iv-automated-deployment/?utm_source=rss&utm_medium=rss&utm_campaign=building-a-sharepoint-20072010-development-environment-part-iv-automated-deployment http://tristanwatkins.com/index.php/building-a-sharepoint-20072010-development-environment-part-iv-automated-deployment/#comments Mon, 16 Nov 2009 06:30:27 +0000 Tristan Watkins http://tristanwatkins.com/?p=505

In the first three parts of this series I covered the project objectives and the system design, then turned my attention to the Hyper-V host image build. In this section I will look at automating deployment of that host operating system. This is lengthy, but there’s a lot to cover.

Before I begin, I need to reiterate a “gotcha” that I stumbled across during the pilot. If you’re thinking of testing this out by adding the Windows Deployment Services role to a machine that’s using Internet Connection Sharing (ICS), think again. They don’t play nicely together. Rather, you’ll be able to deploy the physical machines but any ICS recipient network adapters will be left without connectivity. If you only have one physical machine for testing, consider deploying WDS to a new virtual machine or adding the role within an existing virtual machine.

Imaging

To capture the host OS image we need Windows Deployment Services (WDS), which is a Server Role in Windows Server 2008 R2 (and Windows Server 2008, but we won’t be bothered with earlier incarnations of WDS, RIS or ADS). To specify configuration options during deployment we will attach an unattend file to the captured image. We’ll create it using the WAIK for Windows 7, which includes Windows System Image Manager (Windows SIM).

Preparation

Before we begin, note that WDS needs to be deployed in an Active Directory domain (as a member or a DC) with DHCP and DNS.

• To install WDS, add the Server Role through the Role Summary pane in Server Manager
  • Once launched, configuration options can be selected for WDS, such as default naming patterns and default unattend files. You will not be able to commit any configuration changes unless they are made by a Domain Administrator. If this is a test environment and you are unable to get Domain Admin rights, WDS can still be used with a bit more of a hands-on approach (manually naming servers and joining to the domain, etc.). These configuration options are covered in more detail below
• To download the Windows Automated Installation Kit for Windows 7 (WAIK), visit microsoft.com and grab the latest version
  • Mount the ISO file and install Deployment Imaging Servicing and Management (DISM), which includes the Windows System Image Manager (SIM)
    • A good ISO mounting tool is Magic ISO Maker. Daemon Tools jumped the shark
      
• To use WAIK, you will need to have access to Windows Server 2008 R2 installation media
  • If the server is a virtual machine, consider attaching ISO files of the installation media as optical drives on the IDE controllers in the virtual machine’s settings, as this may make things easier
    • I have created two additional optical drives in my WDS virtual machine, with a Windows 7 ISO file mounted in one virtual optical drive and a Windows Server 2008 R2 ISO file mounted in the other. This allows me to maintain unattend files for both operating systems without having to mess about with real optical/removable media
      
• Ensure that both the WDS machine and the soon-to-be-captured Hyper-V host are connected to the network
• If testing WDS on a live network, be certain that there are no other PXE boot servers on the environment before adding the WDS role, as this may result in network installation chaos otherwise

Configure WDS

This guidance covers some of the settings in WDS that are applicable to this use. I don’t touch on all of them. It should go without saying that WDS deployment should be considered in much greater detail than these summary steps and that administrators should familiarise themselves with WDS and WAIK before unleashing them. That said…

To configure WDS, log on to the server as a domain administrator and launch Server Manager. Expand the Windows Deployment Services node in “Roles”. Right-click the server that you want to configure and select “Properties”.

PXE Response

Choose how strictly you want to control deployment of images. If this is a test environment, configure the role for known and unknown devices. Ideally this configuration is a bit more considered for production. Note: this is distinct from the PXE Boot Policy, which governs the behaviour of the PXE boot process rather than restricting which devices can be booted in to WDS. Think of this as the device setting and PXE Boot Policy as the user setting.

Client Naming Policy

These settings control the auto-generated naming format. For instance, “Dev-%8Username-%02#”, breaks down as follows:

• “Dev-”
• “%8” specifies an eight-character variable
• “Username” is the variable, meaning that the first eight characters of the user’s name will be used in the computer name
• “-” adds a hyphen
• “%02#” specifies a two-character auto-incremented number at the end of the string

Computer Account Location

Specify an OU where new instances of the image should be created.

PXE Boot Policy

Configure this setting to preference. It controls the behaviour of the PXE boot once it has been initiated (either from a one-time boot menu or BIOS boot order change). Once in the PXE boot a DHCP lease is acquired, followed by the behaviour defined here. PXE can either abort and continue with the next BIOS boot option, automatically proceed or prompt the user to press F12 to confirm PXE boot. Think of this as the user setting and PXE Response as the device setting. If unsure what to choose, prompting is probably the best option to choose.

Default Boot Image and Default Unattend file

To simplify things we attached an unattend file directly to the captured image, as spelled out in John Howard’s, slipstreamed Hyper-V article. The unattend file doesn’t exist yet, as we haven’t delved into WAIK yet. For now it’s sufficient to know that these WDS configuration settings determine distinct boot images and default unattend files for different processor architectures. Since we’re attaching the unattend file to the image directly we don’t need to worry about providing values for these settings.

It’s worth reviewing the overlapping options in WAIK and WDS for these defaults settings, as it’s far from straight-forward. TechNet covers this beyond the scope of this article.

Joining a Domain

This setting allows for the newly deployed machines to be automatically joined to the domain.

Network settings

Specify DHCP, Multicast, Transfer Settings (throttling), domain discovery and the UDP port range. Default settings should suffice in a test environment. Approach these choices with care if this environment contains any valuable resources beyond these test machines.

Prepare the Capture Image in WDS

The Capture Image is a PXE-booted version of WinPE specifically designed for capturing custom install images. When PXE booting to capture an image, make sure to select this image rather than a boot image.

• Mount an ISO or insert install media for Windows Server 2008 R2
  • As mentioned above, it might be easiest to add the ISO as a DVD drive on the VM’s IDE Controller. This will also be useful for WAIK configuration later
• Add the Boot WIM to the Boot Images directory
  • Right-click Boot Images in WDS and select “Add Boot Image”
  • Make sure to select the right CPU architecture for the boot image
    • Note: if you forget to do this, you can capture an x64 OS with an x86 Capture Image, but it may be slower to do so. Also, an x64 Capture Image will not be bootable unless the machine has an x64 processor
• Create a Capture Image from the Boot Image
  • Right-click the newly added boot image in WDS and select “Create Capture Image”
  • This will launch a wizard with a link to a help file if the wizard is unclear
  • This is necessary for creating a custom install image in WDS

Generalise the host build

We’re finally ready to get back to the host build. Note: if the host has been built as a virtual machine (as recommended at the top of part III of this series), take a snapshot of the virtual machine before running SysPrep so that the reseal count is not incremented for future updates to the build.

To generalise the host build, run sysprep /oobe /generalize /shutdown. This can also be done from the GUI by double-clicking sysprep.exe.

Capture the install image

Start the host machine and invoke the one-time boot menu (commonly F12).

• Select PXE, NIC or network boot (terminology varies)
  • Depending on the configuration of the Windows Deployment Services (WDS) server it may be necessary to press F12 again to confirm PXE boot
    • If this isn’t done, the machine will boot from hard disk and the machine will need to be SysPreped again
• Select the Capture Image that was created above
  • Again, remember to select the correct processor architecture
• In the Windows Deployment Services Image Capture wizard, choose a location to save the new image
  • This location can be the same disk that’s being captured. I suggest creating a folder called D:\Captured Image so that the directory can be removed with the post-deployment script below
    • Note: the drive letter will not be “C:”, as that’s the WinPE Capture Image that we’re “in” at this point. The System disk will probably appear as “D:”
  • Allow time for the image to be captured locally and then for it to be transfered to WDS
• In my experience, the captured image does not appear in WDS immediately. I’ve needed to re-launch Server Manager to view it and in some cases a reboot was necessary
• If for some reason the network transfer of the image to WDS fails, it can be manually copied to the WDS server and added, but make sure to reboot the server as described above before taking this step or the images may be duplicated and this could get confusing

Creating the Unattend file

Open the Windows System Image Manager from the Start > All Programs > Microsoft Windows AIK menu.

• In the Distribution Share pane, select or create a distribution share in the location of your choice. I created mine in D:\WAIK\DistributionShare
• In the Windows Image pane, choose Select Windows Image and choose the Windows Server 2008 R2 \sources\install.wim file from install media. Select the Standard Full edition when prompted
• Paste the XML below in to a text file and save it as DevBuild.XML
• In the Answer File pane, open DevBuild.XML below as a starting point for your selections

DevBuild.XML

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
 <servicing>
 <package action="configure">
 <assemblyIdentity name="Microsoft-Windows-Foundation-Package" version="6.1.7600.16385" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="" />
 <selection name="Microsoft-Hyper-V" state="true" />
 <selection name="VmHostAgent" state="true" />
 <selection name="Microsoft-Hyper-V-Management-Clients" state="true" />
 <selection name="WirelessNetworking" state="true" />
 <selection name="CoreFileServer" state="true" />
 <selection name="SearchEngine-Server-Package" state="true" />
 </package>
 </servicing>
 <settings pass="windowsPE">
 <component name="Microsoft-Windows-International-Core-WinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <SetupUILanguage>
 <UILanguage>en-GB</UILanguage>
 </SetupUILanguage>
 <InputLocale>en-GB</InputLocale>
 <SystemLocale>en-GB</SystemLocale>
 <UILanguage>en-GB</UILanguage>
 <UserLocale>en-GB</UserLocale>
 </component>
 <component name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <DiskConfiguration>
 <WillShowUI>OnError</WillShowUI>
 </DiskConfiguration>
 <Display>
 <ColorDepth>32</ColorDepth>
 <HorizontalResolution>1280</HorizontalResolution>
 <RefreshRate>60</RefreshRate>
 <VerticalResolution>800</VerticalResolution>
 </Display>
 <ImageInstall>
 <OSImage>
 <InstallToAvailablePartition>true</InstallToAvailablePartition>
 <WillShowUI>OnError</WillShowUI>
 </OSImage>
 </ImageInstall>
 <UserData>
 <ProductKey>
 <WillShowUI>OnError</WillShowUI>
 <Key>XXXXX-XXXXX-XXXXX-XXXXX-XXXXX</Key>
 </ProductKey>
 </UserData>
 </component>
 </settings>
 <settings pass="specialize">
 <component name="Microsoft-Windows-IE-ESC" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <IEHardenAdmin>false</IEHardenAdmin>
 <IEHardenUser>true</IEHardenUser>
 </component>
 <component name="Microsoft-Windows-IE-InternetExplorer" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <CompanyName>Your Company</CompanyName>
 <DisableAccelerators>false</DisableAccelerators>
 <Home_Page>http://yourhomepage.com</Home_Page>
 <IEWelcomeMsg>true</IEWelcomeMsg>
 <SuggestedSitesEnabled>false</SuggestedSitesEnabled>
 </component>
 </settings>
 <settings pass="oobeSystem">
 <component name="Microsoft-Windows-Deployment" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <Reseal>
 <ForceShutdownNow>false</ForceShutdownNow>
 </Reseal>
 </component>
 <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <OOBE>
 <HideEULAPage>true</HideEULAPage>
 <ProtectYourPC>3</ProtectYourPC>
 </OOBE>
 <StartPanelLinks>
 <Link0>C:\Program Files (x86)\Mozilla Firefox\firefox.exe</Link0>
 <Link2>C:\Program Files (x86)\Safari\Safari.exe</Link2>
 <Link1>C:\Program Files (x86)\Opera 10 Beta\opera.exe</Link1>
 </StartPanelLinks>
 <TaskbarLinks>
 <Link0>%windir%\system32\notepad.exe</Link0>
 <Link1>C:\Program Files (x86)\code4ward\Royal TS\</Link1>
 <Link2>C:\Program Files (x86)\Internet Explorer\iexplore.exe</Link2>
 </TaskbarLinks>
 <UserAccounts>
 <AdministratorPassword>
 <Value>encrypted</Value>
 <PlainText>false</PlainText>
 </AdministratorPassword>
 <LocalAccounts>
 <LocalAccount wcm:action="add">
 <Password>
 <Value>encrypted</Value>
 <PlainText>false</PlainText>
 </Password>
 <DisplayName>SourceShare</DisplayName>
 <Name>SourceShare</Name>
 </LocalAccount>
 </LocalAccounts>
 </UserAccounts>
 <TimeZone>GMT Standard Time</TimeZone>
 </component>
 </settings>
 <cpi:offlineImage cpi:source="wim:z:/sources/install.wim#Windows Server 2008 R2 SERVERSTANDARD" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>

Make changes to this file as needed for your environment and save the modified file. I highly recommend reviewing these selections for suitability in your environment. Some sections like the license key and passwords (bolded above) have been replaced with nonsense values. It’s also worth reviewing the options that I haven’t selected. The Windows SIM Help documentation is excellent and it should be a quick learning curve.

Attach the Unattend file to the image in WDS

In the WDS Install Images node, expand the relevant Image Group, find the captured image and right-click to select the Properties pop-up. At the bottom of the General tab there’s a tick box for allowing the image to install in unattended mode and a file selection button for picking the unattend file created above. This will apply the Unattend settings to our captured, sysprep’d image when it’s deployed.

Create the Post-deployment PowerShell configuration script

Before I begin this section, I have to disclaim that my scripting skills are fairly remedial. That said, these scripts have done the trick for us and have saved our support team a good deal of time reconfiguring environments manually.

These steps are necessary because only certain things can be specified in the System Image Manager. For instance, it is possible to configure network adapters and it is possible to add Hyper-V but it is not possible to add Hyper-V, create Hyper-V networks and then configure IP properties for the host’s new network adapters on those Hyper-V networks. To this end, the unattend file above specifies only the addition of the Hyper-V role and this script automates the next two steps. Additionally, this script will create the Source files share on the host machine (as discussed in part II of this series) and remove the local “Captured Image” directory that was created when WDS captured the image to the local drive.

• Create a folder called “Automate” or similar
• Visit the PowerShell Management Library for Hyper-V Codeplex page and download HyperV.zip
  • Run the installer. The script below needs to run from the same directory as hyperv.format.ps1xml and hyperv.ps1
• In the “Automate” folder create a batch script called CreateSourceShare.bat, as follows:

c:
MKDIR C:\Source
NET SHARE Source=C:\Source /GRANT:"Authenticated Users",FULL
ICACLS C:\Source /GRANT SourceShare:F
RMDIR "c:\Captured Image"

• Create a PowerShell script called HyperVNICAndShareScript.ps1 with these contents:
  

netsh interface set interface name="Local Area Connection" newname="Virtual Network Switch"
. .\hyperV.ps1
New-VMExternalSwitch "Hyper-V External Network" -ext "Broadcom"
Start-Sleep -s 5
New-VMInternalSwitch "Hyper-V Internal Network"
Start-Sleep -s 5
New-VMInternalSwitch "Hyper-V ICS Network"
Start-Sleep -s 5
netsh interface set interface name="Local Area Connection" newname="External Network Connection"
netsh interface set interface name="Local Area Connection 3" newname="Internal Network Connection"
netsh interface set interface name="Local Area Connection 4" newname="ICS Network Connection"
netsh interface ipv4 set address name="Internal Network Connection" source=static address=192.168.200.1 mask=255.255.255.0
C:\Automate\CreateSourceShare.bat

Deployment

At long last we are ready to test deployment of the captured image to a target machine. Start the host machine and invoke the one-time boot menu (commonly F12).

• Select PXE, NIC or network boot (terminology varies)
  • Depending on the configuration of the Windows Deployment Services (WDS) server it may be necessary to press F12 again to confirm PXE boot
• Select the x64 boot image for Windows Server 2008 R2
• The WinPE installation process will be nearly identical to installation from media – just make sure to select the captured image when prompted
• Once the installation has completed, run PowerShell as Administrator and set the execution policy to allow remote-signed scripts. For more information see Technet:

Set-ExecutionPolicy RemoteSigned

• Press “Y” to accept the policy change
• Run HyperVNICAndShareScript.ps1
• Confirm that settings match the values assigned in the scripts

Manual post-deployment configuration tasks

Various settings

It’s possible that some of these settings can be automated, but we opted to configure them manually rather than spending time testing the automation of them. They will all be revisited in a later phase.

• Make sure that the BIOS has Hardware Assisted Virtualisation and Data Execution Prevention enabled
• Turn on Write-caching and turn off Write-cache Buffer Flushing on the physical disk’s hardware  policy
• Set Hyper-V default locations as desired. I have chosen:
  C:\Hyper-V\Virtual Hard Disks
  C:\Hyper-V\Virtual Machines
• Disable IPv6 on the network adapters. SharePoint in Hyper-V guidance suggests this is a good idea, as I’ve commented on before
• Set up Internet Connection Sharing from the host’s external network connection to the host’s ICS network connection. By default we share HTTP, HTTPS and RDP

Drivers

We reduced our WDS/WAIK testing time by skipping driver installation automation. As a work-around we are capturing the host image with driver installer files included so that they can be quickly installed post-deployment. We will be introducing automated deployment in a future phase.

Role-based Authorisation and local administration rights

It may also be desirable or necessary to configure role-based authorisation for Hyper-V. John Howard has produced excellent guidance on the model in five parts. This is a link to the fifth. If this is not a requirement then don’t forget to make the recipient of the build a local administrator, as this will be required if role-based authorisation is not configured.

Import virtual machines

As a part of a routine deployment the base virtual machine would be imported, booted up and tested before it’s handed over to the recipient, but since we’ve not built that machine yet, I’m skipping that step for now. The virtual machine build will follow in part five of this series.

Having agreed the project objectives and designed the system, I turned my attention to the Hyper-V host image build. This is a high-level build guide with start-up time and baseline memory consumption benchmarks at key milestones. These benchmark figures were taken from the Windows Server 2008 R2 Release Candidate build and are admittedly a bit imprecise. However, they do provide an overall indication of system performance as things were added to and removed from the installation. Although I do not have precise figures on RTM improvements, I spot-checked a few of these benchmarks when I rebuilt the system on RTM. Start-up times improved slightly at each milestone. In fact, the final benchmarks came in at 100MB less idle memory used in the RTM release.

Capturing a virtual image

For the pilot and the benchmark figures listed below, the image was built on hardware and captured with Windows Deployment Services (WDS). Since then, we’ve rebuilt the system as a Hyper-V guest and we’re capturing the image by PXE booting the SysPrep’d virtual machine from a legacy network adapter. We take a snapshot before SysPrep as a rollback point for future updates to the image. Virtual PC Guy explains this approach in more detail in his geeking out with WDS post post, which I’ve linked here previously. I’ll also be discussing the Windows Server 2008 R2 WDS build and the new version of WAIK in more detail in the next post in this series.

System specification

Dell XPS M1330

• Intel Core 2 Duo T8100 2.1 GHz
• 4.00 GB 667 MHz DDR2 SDRAM
• 7200 RPM 320GB SATA HDD
• nVidia GeForce 8400M GS graphics card
• Windows Server 2008 R2 Build 7100 (RC)

Host build

Initial OS optimisations

I installed the system from a USB pen drive (I had to try this given how easy it is now). On first login I started optimising the system as follows:

• Write-caching is enabled in the physical hard disk hardware policy
  • Buffer flushing is turned off
  • We are comfortable with this risk since laptops run from battery or with battery backup at all times
• IPv6 is turned off on host network adapters per SharePoint’s Hyper-V performance and capacity guidance
  • This is optional, and should not be turned off if you have IPv6 requirements
• IE Enhanced Security Configuration (ESC) is turned off for administrators
  • Again, depending on security requirements, this may not be acceptable
• The following services are disabled (all optional):
  • Certificate Propagation
  • Desktop Window Manager Session Manager
    • Leave this enabled if using Aero Glass
    • In our pilot experience, we identified that Aero Glass suffers from Hyper-V graphics performance degradation enough that it worsens the overall experience
  • IP Helper
    • Leave this enabled if using IPv6
  • Remote Registry
  • Windows Remote Management (WS-Management)

For more information on Windows service hardening, see Black Viper’s site. This list of services could possibly be extended, but at the time of testing service hardening information for the new operating system was harder to come by than it is presently. It should be noted that the Windows Server model of turning services on if needed means that a new system is already considerably harder than a new Windows 7 system.

Next I booted the system, logged in and let it rest for at least fifteen minutes to capture the idle memory benchmark.

Benchmarks

Memory benchmark: as low as 525 MB idle.

Start up time to logon: 37 seconds.

Time to desktop: 52 seconds.

BIOS settings

I installed BIOS A15 (includes alleged performance improvements) and made the following BIOS modifications:

• NIC settings = Enabled with PXE (required for image capture)
• HDD Acoustic Mode = Performance
• Virtualization = Enabled
• Internal Cellular = Off
• Wireless Switch = Wi-Fi + BT

Note: if you have internal cellular and use it, don’t turn it off in the BIOS and adjust the last setting accordingly. Also note, BIOS A14+ is required if exceeding 4GB RAM.

Next I booted the system, logged in and let it rest for at least fifteen minutes to capture the idle memory benchmark.

Benchmarks

Memory benchmark: as low as 525 MB idle.

Start up time to logon: 37 seconds.

Time to desktop: 48 seconds.

Adding drivers and features

Drivers

• SigmaTel Audio driver
• UPEK driver for TouchChip fingerprint Coprocessor (WBF advanced mode)
• nVidia GeForce 8400M GS driver
  • Note: on one occasion this driver install forced a reboot despite requesting that it be postponed – be careful
• Intel and Ricoh chipset drivers (Intel may not be necessary)
• There is still one device without a driver:
  • BCM2045 (Bluetooth)
  • This is because Windows Server 2008+ does not have the Bluetooth stack

Features

• Enabled the desktop experience and WLAN featurs
  • Requires .NET Framework 3.5 and Ink and Handwriting support

Next I rebooted the system to complete the driver installations. On reboot, I set the Windows Audio service to Automatic and started it up. I tested Audio and the wireless network adapter.

Install Windows Search

I chose not to index any drives. This is primarily installed so that Outlook Search can be turned on with little hassle if desired

Other OS modifications

Disabled Shutdown Event Tracker, as developers seem to hate it and it isn’t really necessary on a laptop

Following these changes I rebooted, logged in again and let the system rest for an extended period so that I could capture the next idle memory benchmark.

Benchmarks

Memory benchmark: as low as 622 MB idle.

Start up time to logon: 50 seconds.

Time to desktop: 1 minute  12 seconds.

Installing Microsoft Office client tools

• Office 2007 (defaults, including InfoPath and OneNote)
• Visio 2007
• Office Communicator 2007 R2
• Patched current

Benchmark with Office turned off and Communicator closed

Memory benchmark: as low as 708 MB idle when first launched, idling at closer to 760 MB after a couple of minutes. Proper benchmarks after lengthy idle were not captured here unfortunately. The next set are more useful/reliable.

Start up time to logon: 56 seconds.

Time to desktop: 1 minute 18 seconds.

Installing other client software

I installed the latest versions of all majors browsers, JRE 6, PDF X-Change Reader, Royal TS, browser plugins like Flash/Silverlight and browser debugging tools like Fiddler and Firebug. Some users of the build have specific license requirements that are accommodated manually after the system is deployed.

Benchmarks

Memory benchmark: as low as 740 MB idle.

Start up time to logon: 56 seconds.

Time to desktop: 1 minute 20 seconds.

Summary

That’s it! We met our stated goal to keep the build as lightweight and uncluttered as possible in order to improve performance and reduce the support burden of laptop rebuilds. This system is not much more than a hypervisor, office client applications, web debugging tools and browsers. Keep in mind that I have been building an image that will be captured by WDS. A number of settings will be applied with an unattend file and some additional configuration will be taken care of by script, or manually post-deployment.

Acute observers will note that I haven’t actually added the Hyper-V role. That’s because Hyper-V doesn’t get along well with SysPrep. We account for that with WAIK and scripting, as revealed in more detail in my next post.

Why we’ve chosen Hyper-V

There are broadly five decisive factors: performance, management features (like snapshots), cost, 64-bit OS support and a full host OS (not just a virtualisation administration console):

• Hyper-V is now in its second iteration and has proved to be one of the best-performing virtualisation technologies on the market. With laptops we need to take advantage of every performance gain that we can find.
• Perhaps most importantly, and most often overlooked, Hyper-V is free if you’re already running Windows Server 2008 or Windows Server 2008 R2as a client OS
  • While I work for a Microsoft Gold Partner and Hyper-V is a key part of Microsoft’s future strategy (full disclosure), I believe that most SharePoint developers will have at least an MSDN subscription and a license for a Windows Server operating system, so I believe this is compelling for most SharePoint development environments
• Hyper-V is one of the few virtualisation technologies that supports 64-bit guest operating systems and within that narrow range of choices, it is one of the few that allows a user to log on to a host machine and control virtual machines within it concurrently
  • For instance, VMware ESXi is also a high-performance Type-1 hypervisor that supports 64-bit operating systems, but there is no host machine other than a virtualisation administration console
    • It also has associated costs (support, management tools, etc) that we would prefer to avoid
  • With Windows Server 2008 R2 and Hyper-V, developers can use client tools within a familiar operating system while accessing virtual development environments at the same time
    • This is also true of VMware Workstation, but it’s a Type-2 hypervisor and will have relatively poor performance
  • We need to have 64-bit OS support for Windows Server 2008 R2 and SharePoint 2010 – both of which do not have 32-bit flavors

Our approach

Workgroup development

By building our virtual machines in a Workgroup, we no longer need to worry about SharePoint installation/(re)configuration difficulties, as we will import the same identical virtual machine on everyone’s Hyper-V host. This would not be possible if the virtual machine was a member of a centralised domain because the domain controller would receive chatter from many identical machines and everything would quickly start to unravel

Alternately, we could run Active Directory Domain Services within a virtual machine, but this has a performance overhead that is best avoided unless it is absolutely necessary. Additionally, developing on a domain controller is not ideal

While developing in a Workgroup presents challenges for profile imports, these are far from insurmountable when LDAP directories like AD LDS or SQL users can substitute in many scenarios. The only evident need is a scenario where Active Directory Domain Services (more than just user accounts) are required, and that’s certainly not going to be true of enough projects that it should form a part of the base build. To this end, as requirements for Domain Services are identified we will provide new builds, as they are likely to be very project-specific

Networking

Internet Connection Sharing network

In order to isolate identical virtual machines from each other and from network resources, I’ve created a Hyper-V internal network dedicated to receiving Internet Connection Sharing (ICS) from one of the host’s active network connections. This is an internal network like any other Hyper-V internal network – it just so happens that the host’s adapter on this network will be receiving ICS from another of the host’s connected adapters. Any connection can share to the ICS network – even if Hyper-V doesn’t natively support external networks of that type. Depending on the need, we will share to this ICS network from:

• Hyper-V host external connections (by default)
• Wireless
• Mobile broadband
• VPN

ICS also introduces a layer of NAT between the guests and the physical network, preventing inbound connections to guests over these networks. This is desirable as it is how we achieve physical network isolation, and is the reason why we’ve chosen ICS over Bridging. In the ICS Settings we enable outbound RDP, HTTP and HTTPS connections over ICS by default, although it may be useful to enable other common outbound network protocols like FTP and SMTP. Outbound connectivity from our guest virtual machines is primarily used for connecting to TFS over HTTPS.

Internal network

We have also created a Hyper-V Internal network to support “always on” communication between the host machine and the guest virtual machines. Guest virtual machines can also communicate with each other over this network.

We cannot rely solely on the ICS connection. Guest virtual machine IP addresses on the ICS network will change because they receive them via DHCP from the host’s ICS connection. This is just how ICS works. The host’s ICS adapter becomes a gateway on 192.168.137.1. Any Hyper-V guests on that network pick up DHCP from the host and are automatically assigned an address on the 192.168.137.xxx (255.255.255.0) IP range.

As we rely on HOSTS file entries for RDP connections from host to guest and for browsing to guest SharePoint sites from the host, we need fixed, reliable IP routing and name resolution, so we use this second network for that purpose. The hosts and guests have both been built with fixed IP addresses on this range.

Because this is a Hyper-V internal network, there is no risk of network collisions on these IP ranges (which are identical for all users). Internal traffic never leaves the machine.

Project builds

By providing self-contained environments, our technical leads and/or architects can now customise the base builds to create project-specific virtual machines that can be exported to all members of a team, reducing system/configuration inconsistencies. In practice, the build lead/architect will export the final snapshot of the project environment, which all team members will use as a starting point for project development. As the project progresses, updated builds can be released in this same manner.

Clean hosts

Host machines will be cleaned of development tools and data, allowing quick provisioning. Development tools will not need to be reconfigured, as they will reside in virtual machines that can be exported/imported to any host. Guest base build virtual machines will be provided with as much SharePoint configuration as possible in order to make them light on reconfiguration, disposable and re-deployable.

Project resumption

Project-specific exports will reduce the need for storage of multiple virtual machines – as much as possible. We will retain one virtual machine export in public storage per-project. This will reduce the amount of time involved with resurrecting environments after project completion.

Optimisation

Host machines are built on Windows Server 2008 R2, optimised as much as possible and reduced to the lightest weight achievable. The build will broadly include:

• Hyper-V R2
• Microsoft Office applications
• All browsers
• No development tools (these will all live in guest machines)
  • The only exception to this is the Team Foundation Client which TFS administrators manually install. We have not been able to pick domain users from within a Workgroup environment

Content provisioning

By deploying a single project-specific virtual machine, we can bake content in to the project build, ensuring consistency and reducing the overhead associated with re-deploying content.

Improved testing and reduced volatility through the use of snapshots

By using snapshots we are able to test code and configuration changes without volatility. The benefits of this technology include:

• Capturing restore points at milestones in a server build
• Capturing a stable state before attempting volatile configuration changes
• Capturing a stable state before testing code changes
• Creating an initial restore point after importing a virtual machine and re-configuring network adapters (when required) and making other preferential settings
  • This saves re-importing and reconfiguring network adapters if a machine needs to be rolled back to its initial state
• Exporting a virtual machine to capture a problem when trouble occurs
  • An exact instance of a problem can be captured and shipped out for support, without having to re-create the problem in a distinct environment. This will only apply tot self-contained environments, however

Local source code storage on the host machine

Before our pilot started we identified that storing the local copies of source code within changing snapshot states could create problems. At the same time, we found it desirable to put the development tools inside our virtual machines in order to get around remote SharePoint development difficulties and to keep the host build uncluttered. To resolve this conflict, we adopted this approach:

• Created a share on the host machine and granted ownership of that directory to a new user account that is used solely for this purpose
• Created a new user account with the same name and password in the guest virtual machine
• Mapped a drive from the guest to the host’s new share, using the internal network’s IP address and these new credentials
• Launched Visual Studio and downloaded project source code, pointing to the newly mapped drive as the location for the local source
• Created a new Code Group for the mapped drive to enable trust for code execution

These steps are covered in more detail in the build guides that will follow in later posts. This is just an outline of the approach to extracting the local code from the snapshot state, which should be adaptable to other development systems with snapshots. For instance, we’ve found it necessary to download code to a unique location within this directory for each each user, as TFS tracks the network locations that code is checked out to. This is easy to get around, as each user just specifies a unique directory name. If this were insufficient for project requirements, this TFS behaviour could possibly be “fooled” by using NTFS junction points. We’ve not seen a need for this yet, but we’re confident that with this additional option we should be able to store local source code in this manner, and this has been validated by our project experience with Hyper-V to-date.

Summary

These are the high-level design choices that emerged early in the consultancy and research, which have remained largely unchanged to-date. These choices represent one design that has been validated for our needs, which has some shortcomings like any approach. Some of these issues will be covered in the final post in this series.

The information in this post has not covered implementation in any detail, but do not fret. In the next section I will cover the step-by-step build guide for the Hyper-V host laptop.

Historical development approaches

We have used a variety of development approaches in the past, which have all had limitations and were often an obstacle to productivity. They included:

Shared development farms

Shared farms are appealing at face value because they put development resource hunger on beefier infrastructure. However, there are considerable drawbacks to this approach, including:

• The infrastructure burden of these environments is immense
• Due to limited physical resources, these servers housed multiple projects in a single farm, leading to hive pollution and a less reliable/predictable development environment
• When multiple developers would use the same farm, this would amplify resource contention on already strained servers and it made managing application-level change much more difficult

Local development

Developing SharePoint directly on a Windows Server 2008 laptop introduced a need for frequent laptop rebuilds, as the SharePoint customisations for one project rarely carry over to another and the hive gets polluted. This adds a hefty support burden and frequent down-time for developers.

Virtualised development using Virtual PC

Virtual PC is a friendly technology for SharePoint developers to use, as it does not over-complicate virtualisation, but the management functionality (such as snapshots and snapshot export) is inflexible and performance is poor when compared to a Type-1 hypervisor.

Third-party virtualisation technologies

Most third-party technologies have associated license costs for businesses and many do not support virtualisation of 64-bit guest operating systems. This is problematic because:

• Windows Server 2008 R2 is only available in 64-bit
• SharePoint 2010 is only available in 64-bit

Additionally, many developers have strong preferences for third-party virtualisation vendors and if this is not controlled/standardised it can introduce incompatibility issues.

Why none of these approaches were satisfactory

In short, these environments did not offer good enough performance or management flexibility and we needed a standardised approach for business continuity. We also needed a technology that would support 64-bit systems.

SharePoint development complications

SharePoint development in teams has always been tricky, as SharePoint virtual machines cannot be easily cloned and renamed. This means that SharePoint environments require:

• Scripted installation, which is great for standard builds, but not terribly easy to reconfigure for project-specific requirements, or…
• Manual installation, which is time-consuming and not always aligned with a developer’s skill set, or…
• Shared environments, which suffer from the drawbacks outlined above, or…
• Network isolation of the cloned machine so that that nothing on the network will ever be aware that there are clones

These are some of the biggest issues that need to be confronted when designing a SharePoint development environment. This list is by no means exhaustive; in the final post in this series I’ll be reviewing some of the issues that we’ve encountered during a pilot and subsequent roll-out to all of our developers, consultants and architects.

Distilled objectives

After a thorough review of development approaches, difficulties and available technologies, a Windows Server 2008 R2 (RC) with Hyper-V pilot was announced, soliciting senior developer/architect participation, stating the following aims:

• Improve performance over Virtual PC by adoption of a new core virtualisation technology
• All available performance guidance for hardware, Windows, IIS, SQL and SharePoint would be considered for inclusion in the build
• Hyper-V R2 was adopted early for the pilot (at Release Candidate) to take advantage of further performance gains in the new version
• Project cost effectiveness would be improved by:
  • Removing development tools and SharePoint from the host machine, reducing the amount of reconfiguration on laptop rebuild and reducing rebuild requests
  • Providing a suite of pre-configured virtual environments, reducing setup time at project commencement
  • Making project-configured environments reusable within teams throughout the life of the project with Hyper-V’s snapshot technology
  • Eliminating the infrastructure burden of shared development farms and re-provisioning those resources to better purpose
• Improve the testing process by introducing snapshots

In conjunction with the pilot we upgraded physical disk speed and capacity from 160GB 5400 RPM drives to 300GB 7200 RPM, as Hyper-V snapshots chew up lots of space, and the added spindle speed is critical to improving performance.

Defining these objectives, limiting the duration of the pilot and testing the new approach were critical to gaining management support for the project. In advance of pilot deployment I trained all pilot participants on Hyper-V and introduced our approach to networking and source code storage, which would not be obvious otherwise.

In the next section I will cover the design decisions that were reached through consultancy with the business, pilot findings and research.

To help pin this down, I will add that this Hyper-V host machine has been deployed from a WDS image, but I initially discounted that as an issue because the Hyper-V role is not part of the image; it is added during the deployment as specified in an unattended installation file and the Hyper-v networks are created by script post-deployment. Since the Hyper-V host’s MAC address is distinct from the physical adapter’s address, I’ve been assuming that there must be an algorithm that generates the host’s virtual MAC address uniquely, but given that this problem seems to manifest itself soon after installation I’m leaving this possibility on the table.

I’ve noticed a fair amount of traffic for this topic. If you’re having the same problem and this suggestion fixes it, I’d be grateful to know if the Hyper-V system was deployed from an image, as that might help to narrow the troubleshooting effort.