I’ve recently persuaded myself not to purchase a new laptop, trying to get another year out of my HP Compaq nx7300 (running 32-bit Windows XP), which I purchased about four years ago. My home computing needs are fairly meagre, but I recently spent £23 upgrading my RAM to 4GB to see if I could get a strained instance of my SharePoint 2010 development environment running at home as needed. To this end, I decided I would clear enough System drive space to make room for Ubuntu (x64) and run my VMs from Virtual Box in the other partition, in order to get the most memory possible to my VMs. Like I said, strained…

So I loaded up the Ubuntu installation CD and launched GParted, cleared ~10GB for the OS and another 5GB for the SWAP partition, then installed. All went well, I installed VirtualBox, rebooted in to XP to make sure everything was cool over there. Went back in to Ubuntu to get a bit more familiar with the new version then shut down for the night. The next evening when I started up I was confronted with an ominous message:

No module name found
Aborted. Press any key to exit.

Non-System disk or disk error.

Not good. I checked I was booting to the right drive, then ran the Ubuntu installation disk again, which includes a no-installation trial option that runs entirely from CD/memory. I loaded up GParted again, checked all of the partitions, didn’t notice anything strange and then started searching for an answer. Luckily I dug a bit deeper than the first few results which recommended a full re-install of both boots. While that might have fixed the problem, it wouldn’t have been very desirable.

Eventually I found this Ubuntu forum thread, Grub “no module name found” after reboot, which points to this (in my opinion) definitive SourceForge article on Boot Problems:Windows Writes To MBR. While I would argue the problem actually has very little to do with Windows, it identifies the cause and four possible solutions. To summarise, the Windows partition contains software that writes to the Master Boot Record, which in turn corrupts GRUB, the Ubuntu bootloader. I say this has little to do with Windows because in all of the cases listed in this article and on the Ubuntu fora, the problem is caused by something that gets beneath Windows, like OEM “protection” tools, for lack of a better description. Here’s the list from SourceForge:

• HP: Credential Manager, Recovery Manager, ProtectTools, PC Angel, Backup and Recovery
• Dell: Recovery Tools, DataSafe Local Backup,
• Samsung: Recovery Solution III
• McAfee Security Center ???

This is all stuff that ships with your machine. I’ve always been distrustful of this software but I never got around to removing the HP tools on this laptop. Eventually I got worried that removing the credential management software might lock me out of the machine. Fairly irrational, I grant you, but as I say, I deeply distrust most things in Windows that appear to be getting beneath it and so I wind up avoiding them, even if they’re just sitting there doing nothing.

Back to the problem at hand, I eventually managed to restore the bootloader by following the first recommended solution from the SourceForge article – I uninstalled all of the HP software (except for Wireless/Bluetooth). However, there was one fiddle that’s worth noting. The HP Backup and Recovery Manager software did not have an entry in Add/Remove Programs and there was no uninstall option in the program or the Start menu. A quick search returned this recommendation, which launched the installer and started removing the tool.

RunDll32 C:\PROGRA~1\COMMON~1\INSTAL~1\PROFES~1\RunTime\10\01\Intel32\Ctor.dll,LaunchSetup "C:\Program Files\InstallShield Installation Information\{3F9F7336-6DF8-476F-ABF6-C70A17FAF619}\setup.exe" -l0x9 -uninst –removeonly

After rebooting into the Ubuntu installation media again I ran the recommended GRUB fix-up.

sudo mount /dev/sda6 /mnt
sudo grub-install --recheck --root-directory=/mnt /dev/sda

As recommended in the posts above, make sure “sda6″ and “sda” refer to the correct targets. Basically, make sure to familiarise yourself with those posts thoroughly before doing anything, and of source I present this without warranty.

After shutting down and starting up again I was presented with the GRUB bootloader as expected. After shutting down one more time, everything continued to work. That’s the key. Until the corrupting software is removed you can repair the GRUB bootloader with the install CD and it will remain fixed until shut down. On the first boot after shut down the MBR changes from the offending software in the Windows partition will be referenced and GRUB will fail to load. This is why the GRUB fix needs to be made after removing the offending software.

Rather irritatingly, I’d just started to doubt my suspicion of OEM software like this when I got to play with the Lenovo W520 tools briefly. They seemed quite useful. Indeed, my colleague Wes Hackett was just saying how much he liked the biometrics on it, so it’s a shame to have my suspicions slapped back in to place through this episode.

SSD

Probably the most contentious finding from my initial testing was that disk performance and bus speed aren’t significant factors in most of those results (start-up and shut-down times being a notable exception). To recap a bit of my initial summary:

Disk performance and bus speed did not prove to be significant factors in these results (except for virtual machine start-up times). Obviously there are fundamental differences about SSD (yet untested) that may skew this picture, but I will be surprised to see big differences. If we’ve got these tests right, and they are actually representative of the tasks that slow down development, then we would expect to see wider variance across bus or disk speeds. We don’t.

• This assumes the disk is relatively uncontended. Virtual machine performance degrades in every type of test while large file operations are running concurrently on the same disk. This could be copying an ISO, importing or exporting a virtual machine or any other sustained large file operations.

The obvious follow-on test would be to repeat on the same system with SSD. Unfortunately I’ve not found the time or hardware resources to do that yet, but today I ran an indicative test. In this scenario, I installed two new boots on a brand new Lenovo ThinkPad W520. One drive was an SSD in the second bay, the other was a 7200 RPM SATA drive (I don’t have specs for either to hand, but they were the default Lenovo offerings). For both boots I ran the VMs on the other spindle, so we had one test with an SSD system drive and VMs running on mechanical drive. For the second test I inverted the configuration and had a mechanical system drive with VMs running on SSD. In both cases there was no appreciable system contention outside of these tests.

The results? Identical. First page load after application pool recycle times were around 10 seconds for Central Administration, a blank site and a My Site Host. 16 seconds for a customised intranet solution (the same one from the initial tests). These are very similar times to the desktop results from my original tests – only marginally slower. What does this tell me? I should complete the testing for more scenarios than just the first page load times. But given that it won’t happen any time soon, I’m pretty comfortable assuming that SSD isn’t going to automagic performance improvements where disk speed is otherwise not a factor, and I’m happy standing by my initial analysis with this supplementary finding in hand.

To be crystal clear, I’ve seen first-hand how quickly a VM starts and shuts down when running on SSD. It’s stunning. And there’s clearly a subsequent gain reaching a post-start-up stasis of sorts. I always waited for my system to calm down like this before any testing could begin, and in some cases that might take ten minutes on a 7200 RPM mechanical drive, and even longer over USB2. However, I don’t actually see this as a major productivity loss. Irritating, yes. A sound business case? Probably less so. I imagine doing lots of full crawls would translate to a big productivity gain on SSD, but is that a major issue for most developers on most projects? Not consistently so, in my experience. But if you’re developing a FAST solution it would probably be a good idea. Maybe even isolate all of the DBs on the SSD. There would certainly be scope to play with this once you have known disk contention that you’re fighting.

The problem I have is that I can’t find any other scenarios which are as disk-bound as we might assume. When we first started this testing in late 2009, our first inclination was to add eSATA drives on a PCI Express port to get a second spindle. Freeing up the VMs from system activity and large file operations on the system disk is a clear win, but this will be true for any disk of any speed on virtually any bus if my initial test results are to be trusted, which means that the SSD investment for VM performance gains is only likely to get you faster start-up/shutdown times and anything else that involves large file operations.

All this said, if budget and SSD reliability are not concerns, load up on them, assuming it gets you sufficient storage capacity. It won’t hurt, so long as they don’t fail all the time. Additionally, it may be beneficial to get an SSD for the system drive, if other non-development activities would benefit from it. Or it may be that start-up/shutdown times are compelling on their own. In the final analysis, I’m in no way opposed to SSD, but when it’s my neck on the line for justifying hardware purchases, I want concrete, consistently-realised performance gains if I’m going to recommend a less resilient, lower capacity, more expensive technology. In most cases, I’m not sure that’s the case for virtualised SharePoint development.

i5 vs. i7

One of the other key follow-on investigations from last year’s testing was a comparison of i5 vs. i7 processors. I’ll quote the initial context here:

• The benefit of spending on i7 processors is in doubt. We are seeing very minor performance penalties when adding more than two CPUs in both VMWare Workstation and Hyper-V for most tests. There were also very minor improvements for some tasks, but on the whole there does not appear to be a measurable benefit. This might vary if the host OS is doing a great deal with the CPU, but that is liable to cause other contention issues than just in the CPU (on a laptop).
• The only tasks that appeared to use all 8 cores in a SharePoint VM were:
  • Retract/Deploy of a solution (but only very briefly)
  • Create web app, or Create site collection (but at low percentages)
  • Rebuild with Code Analysis (but not fully)

Since the initial testing, I’ve continued to experiment with two versus four cores in the VM, and have never seen a significant enough difference to endorse using more than two, but at the same time, I don’t think the penalties for multiple cores are significant enough to worry about, if any user thinks that four cores will be better. Note: I’m only talking about development here.

Based on these findings, I had a hunch that a faster clock speed i5 would outperform an i7, assuming two or fours cores running inside the SharePoint VM. For the sake of simplicity I’ve tested with two cores. For these follow-on tests I used the same ASUS V6-P7H55E model that I used during the original testing, with an identical spec/configuration and the same VM, with one exception. We replaced the Intel® Core™ i7-870 Processor (8M Cache, 2.93 GHz) with an Intel® Core™ i5-680 Processor (4M Cache, 3.60 GHz) – faster speed, smaller cache.

To my surprise, the performance tests returned virtually identical results to my initial testing (all within the margins that the initial tests deviated). Reviewing those results again, we can see that for most tests disk performance is not an issue (see above), and these tests suggest that CPU is not a bottleneck to further performance gains beyond a certain point (I believe an older CPU would fare poorly against either of these, but if a 4.0 GHz i5 came along, I’m not sure we’d see an improvement over these results). These machines have reasonably high-spec RAM, so memory speed does not seem a likely candidate for further improvements. Based on resource monitoring during testing, I can’t see that anything is maxed out, so I’m beginning to think there’s something inherently languid in the sequence of this computation. Perhaps a deeper dive is in order some day, but I’m probably not the best person to take that on.

As an aside, I can confirm that I’ve been running up to six VMs concurrently on this desktop with the i5 over the last couple of weeks. Starting all of the machines up at once is rough, but after 15-20 minutes it’s handling it no problem, and I don’t have to do that unless I’m taking a major snapshot. This suggests disk starts to become an issue with six VMs running at once, but that shouldn’t surprise anyone. If anything I’m surprised it’s not more of an issue on this machine, and if I continue to need this many VMs at once I’ll probably sacrifice my RAID 1 array for two separate disks. I’d be hesitant to suggest SSD in this case, since six VMs is probably going to chew up more storage than most SSDs will accommodate.

Based on these results and this longer-term experience, I’d recommend the higher-speed i5. I don’t seem to lose anything with the i5, at any rate. Maybe even go down to a 3.0 GHz i5 and save some money? If you know you have a specific scenario that will consistently utilise eight cores, go for the i7. But ultimately, both of these CPUs are fast.

Windows Experience Index and CPU Benchmarks

I’ve had a number of discussions with people about performance since publishing these posts, and it’s surprised me to find how many people actually look at the Windows Experience Index. Unfortunately, in my experience, this really doesn’t tell us much on today’s machines. A poor-to-average developer machine today gets a good score, unless it has a 4200 RPM hard drive (in which case it shouldn’t be used by a developer). Also, graphics performance is probably irrelevant. I really don’t think this index sheds any light on the SharePoint Development Experience Index, as it were.

Along these lines, with the receipt of a few new Sandy Bridge CPUs in these Lenovo laptops, we started running CPU benchmark tools. These are quite useful for diagnosing problems (early BIOS versions on the W520 were slooooooooooooooooooow – make sure to apply v1.25+), but beyond that, I’m not sure they tell us what we need to know for SharePoint development. For instance, at one point we saw hugely different CPU benchmark scores but the SharePoint performance tests were roughly the same. I guess I mention these tools here to say that they may be useful in some cases, but I think these real world tests probably tell us more.

Sandy Bridge

“How good are these Sandy Bridge CPUs”, I hear you mutter? Battery life is amazing. I accidentally left the Lenovo W520 running unplugged all day today. I think it lasted about six hours. Performance-wise, you’ve seen the ~10 second first page load times after an application pool recycle on a few of the standard SharePoint OOTB templates. That’s on a 2.0 GHz Sandy Bridge i7. This is not far off the 2.93 GHz first generation i7 desktop CPU results from our original tests, and much better than the first-generation i7 laptop PCU. Pretty good, I’d say. I can’t wait to see the second generation desktop speeds. Note: the desktop i7 models also have integrated graphics, where the first-generation desktop i7 CPUs did not. Now to hope there aren’t any more recall issues.

A few other things to note:

• Until a few months ago, I didn’t realise that dual-core i7 CPUs exist. I thought they were all quad-core. Not so. This is important because if you find a laptop model with four SODIMM slots (to get you 16GB RAM, and 32GB in due course), the fine print will probably tell you that you will only get two SODIMM slots unless you purchase a quad-core CPU.
  • There’s a secondary “gotcha” here, in that the quad-core laptop i7 CPUs peak at a much lower clock rate than their desktop siblings. I think the fastest first-generation quad-core laptop i7 CPU peaks at just over 2 GHz, and most laptop manufacturers have very few models, if any, with this CPU. In fact, we struggled to find anything other than 1 Lenovo, 1 Dell and 1 HP model at 15″. Most of these only had availability for lower clock speeds and we nearly had to settle for 1.73 GHz. These are all very expensive as well.
• The Sandy Bridge comes to the rescue here insofar as it has higher clock speed quad-core laptop i7 models, even if these are also slower than their desktop siblings. However, it’s also worth noting that in most CPU comparisons of Sandy Bridge to 1st-generation i7 models, the Sandy Bridge annihilates. Basically, at this point, if shopping for a high-performance SharePoint development laptop, you should be looking at Sandy Bridge. They may actually be cheaper as well – somehow.
• Also be aware that the Japanese earthquake has caused severe manufacturing delays for most hardware vendors. You may find you need to settle for a lengthy lead time at the moment.

When I learned WDS and the Windows Automated Installation Kit (which were both quite new in Windows Server 2008 R2 at the time), I contented myself with getting ~90% of the way to a fully-automated build, as the additional effort to get from 90 to 100% (mostly re: drivers) wouldn’t have paid enough immediate dividends and we needed to start capitalising on some of the other wins of our new environment. As is often the case, we never got back to that remaining 10%, but it’s become more of an issue in recent months, as we’ve added a few Dell Latitude E6410 and Lenovo W520 laptops – both of which had network drivers that the Windows 7/Windows Server 2008 R2 boot images didn’t recognise. Unfortunately the TechNet guidance on adding drivers to boot images is unclear (to me anyway), so I’m contributing this quick post to attempt to clarify the problem that we had and the simple step-by-step solution.

A matching network card driver was not found in this image

After preparing our image with current patches and making the state as general-purpose as possible, we ran SysPrep with Generalise and OOBE, then Shut Down the machine. I always Shut Down rather than rebooting because I don’t want to miss the window in which I need to hit F12 to trigger the PXE boot to capture the image. If the post-SysPrep boot initiates there’s a risk that the SysPrep rearm count will be incremented, which is rather undesirable.

During capture, I was able to run through the wizard but I was not able to connect to the WDS server during the imaging process. Not the end of the world… I just manually uploaded the image after the process completed. However, this was my first indication that all would not be well with the NIC drivers. Note: my solution below should be repeatable for the capture images as well as the boot images, correcting this issue as well.

When it came time to deploy the image, we got in to the Windows PE setup splash, but no further than this error:

WdsClient: An error occurred while starting networking: a matching network card driver was not found in this image. Please have your Administrator add the network driver for this machine to the Windows PE image on the Windows Deployment Services server.

An Outdated KB

As you will note in this knowledge base article (which dominates search results for this error), the work-around is fairly detailed and laborious. Nevertheless, I proceeded, with a few caveats.

1. I didn’t actually get the error that the KB article describes from the Setupapi.app.log, so after a bit of head scratching, I moved on to step 2, deducing which driver I needed from my extracted NIC driver INF file.
2. peimg /inf=driver.inf mount\Windows, from step 3h, just didn’t work for me. “PEImg” couldn’t be found. Eventually I figured out that PEImg refers to an older version of Windows Deployment Services, so this just didn’t work.

At this point I went back to the drawing board and started reviewing the Windows Server 2008 R2 TechNet documentation, leaving this KB article behind. I was pretty sure there was a less convoluted way of getting this done anyway. Eventually I found the Add Driver Packages to Boot Image Wizard, as I’ll detail in step-by-step instructions below, but now I was getting error code 0xc1420127 in the wizard, as detailed here (with a good screen shot) and here (with this solution):

1. Clear all your temp directorys.
2. Browse to “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WIMMount\Mounted Images” and delete any keys below this.

I think the important step here is the second one, which removes the mounted image that I never unmounted via imagex /unmount /commit mount; the registry keys align precisely with Imagex /info Drive:\remoteinstall\boot\x86\images\boot.wim and Imagex /mountrw D:\remoteinstall\boot\x86\images\boot.wim 2 mountfrom steps 3f/3g/3h:

I purposely avoided committing the mount since I couldn’t make the PEImg changes, but this inadvertently caused the Add Driver Packages to Boot Image Wizard 0xc1420127 error.

A much simpler solution

After deleting these keys, I was back on track. If I’d never stepped through the outdated KB article I could have followed these steps below and saved myself (and apparently a few others) much hassle, but for whatever reason the Add Driver Package command has always eluded me – tucked away as it is under the Drivers node in WDS. I was always distracted by the Add Driver Packages to Image command under the Boot Images node, as in step 3 below, which gets you nowhere without adding the driver first. But once you find that and step through, it’s pretty easy.

1. Add Driver Package
   
2. Select driver packages from an .inf file
   
3. On the boot image, select Add Driver Packages to Image:
   
4. Click Search for Packages:
   
5. While adding the package to the image it will be temporarily dismounted. In order to account for this in advance you can temporarily disable the image before doing any of this and then re-enable it afterwards.

Repeat this process for other boot/capture images as needed, and make sure the driver matches the boot/capture image architecture. The install image doesn’t need to match the boot image architecture though.

Ultimately, this all shows off how much better WDS in Windows Server 2008 R2 is than its predecessors, which were dark arts that few could master. Not so any more, but unfortunately automated deployment is still confusing when it goes wrong  per the number of technologies that all support the same or similar ends, new and old, including WDS, WAIK, MDOP, SCCM, DISM, RIS, ADS and I’ve forgotten how many others, especially when the changing interrelationships between these products over time further obscures the quality of guidance.

Context

A couple of months ago I was building a client’s production farm. It was a pretty straight-forward architecture with few unusual requirements. I’d successfully provisioned everything and was deploying the PDF iFilter as one of my last steps. When I ran a test crawl to see if it could pick up the contents of PDF documents, I was surprised to find the Local SharePoint sites Scope contained zero items, even though the crawl successfully gathered 459 items. To add to my confusion, the People scope was fully populated. I verified that the scope didn’t need to be updated, then launched ULS Viewer. While reading the trace logs in real time, I re-ran a full crawl and spotted this clue (my bold):

AuthzInitializeContextFromSid failed with ERROR_ACCESS_DENIED. This error indicates that the account under which this process is executing may not have read access to the tokenGroupsGlobalAndUniversal attribute on the querying user’s Active Directory object. Query results which require non-Claims Windows authorization will not be returned to this querying user.

Investigation

This error message reveals quite a bit. We know the error occurs in a w3wp.exe process associated with SharePoint Server Search’s, “Query Processor”, and that the application pool identity of this process doesn’t have read access to the tokenGroupsGlobalAndUniversal attribute in Active Directory. This tells us the error is occurring on the SharePoint Search Service Application pool’s identity, rather than on the Search Service (which is not a w3wp). After searching for a bit I found a few useful posts/articles, but what really helped me was Russ Maxwell’s article, which I linked to at the top of this post.

I suspect that in his testing, Russ found different scenarios where Pre-Windows 2000 Compatibility Access rights needed to be granted to the Search service account, but in my case these rights didn’t help. His error and his explanation of the problem are different. I don’t want to make too much of this, since his post was circa Beta, but it’s worth noting there may be multiple issues with these rights and Search. In our case, we tried to grant rights to the Search Service account but the error persisted until we added the Search Service Application Pool Identity account to this group. In actuality, we identified these same errors on the farm account initially as well, but granting these rights to the farm account didn’t solve the problem.

I should also note for completeness, that there were Security event 4625 Logon Failure errors accompanying the ULS log entries until we granted access to the Search Service Application Pool Identity account, at which point these events were replaced by 4624 Success events.

After running one more Full Crawl I confirmed that the ULS errors were also gone. It’s reasonable to infer from these new ULS events that when PluggableSecurityTrimmerManager is selecting, “workid from scope()”, it needs these Pre-Windows 2000 Compatibility Access permissions in a Windows Server 2000 or Windows Server 2003 domain. Presumably if SIDs can’t be initialised, everything gets security trimmed.

How to use these findings

I’d recommend adding this to the list of permissions you may need to grant in a Windows Server 2000 or Windows Server 2003 domain. This is basically what Russ Maxwell was saying initially, as I read it. In this scenario, I’ve merely spotted a scenario where different rights are required and I can’t shed any light on why this hasn’t been required in every Windows 2000 or Windows 2003 domain I’ve worked in.

If working from a principle of least privileged access, I’d suggest granting these rights as needed during deployment. They shouldn’t need to be granted particularly broadly (unless you’re working with a 1-way trust from a resource domain, which is another story – see the comments in the Russ Maxwell post for an introduction). Alternately, it’s arguable that granting read access to this tokenGroupsGlobalAndUniversal (TGGAU) attribute isn’t opening an enormous hole, but that’s a question for each organisation to answer based on their security models.

A Note on User Profile Pre-Windows 2000 Compatibility Access Rights

While I’m speaking of variance in these permission requirements, I should note that I’ve seen a number of sources including Spencer Harbar, TechNet and this Russ Maxwell article mentioning the need to grant these same Pre-Windows 2000 Compatibility Access rights to the User Profile Synchronisation account, but I haven’t had any problems running without these rights in two different Windows 2000 or 2003 domains.

If the domain controller is running Windows Server 2003, the synchronization account must be a member of the Pre-Windows 2000 Compatible Access built-in group. See Add an account to the Pre-Windows 2000 Compatible Access group for instructions to grant this permission.

Given those sources, I’d suggest you’re probably best off granting the rights, but I haven’t yet been able to validate the need myself. I’d definitely be interested if anyone can shed more light on that topic.

Round I

My difficulties started when I attempted to move a newly-provisioned Query Component to a web front end server. When it failed, I tracked the problem down to missing permissions on C:\Windows\Tasks. At this point I didn’t know why the permissions had been removed and this was actually the first time I’d noted these permission requirements. TechNet suggests WSS_ADMIN_WPG needs Full Control of %WINDIR%\Tasks, but the description of this requirement is “N/A”. Oddly, according to this TechNet article, the WSS_WPG group does not appear to need these same rights, although they are assigned by the SharePoint installation/configuration processes – or at least they are in the environments that I’ve built.

Adding to this confusion, I found this strange ULS event, in which the provisioning process tries to remove WSS_WPG access to %WINDIR%\Tasks and grant R/W access to the Search service account. This is pretty weird! It might explain why the WSS_ADMIN_WPG group needs Full Control rather than just R/W access, but I wouldn’t typically expect SharePoint to be modifying ACLs in the Windows directory.

“Modifying ACL to allow R/W access to ‘C:\Windows\Tasks’ and to remove access for WSS_WPG.”

Back to the provisioning problem at hand, once I added the missing permissions for both the WSS_WPG and WSS_ADMIN_WPG local groups on %WINDIR%\Tasks the provisioning process completed successfully. You can also see that the “Modifying ACL” event directly precedes the failure to start the new Service Instance. While this event helped me track down the problem, and is clearly related to it, unfortunately I need to leave that mystery behind for now, as there are bigger issues to address in this post.

Round II

Later, this client got back in touch and mentioned that their Search Service Application wasn’t working. In this case the Search Administration page was available but all Content Sources, Scopes, Crawl Logs, etc. pages failed with errors on the Admin Component.

Crawl status: The search service is not able to connect to the machine that hosts the administration component. Verify that the administration component <GUID> in search application ‘<Search Service Application name>’ is in a good state and try again.

To cut a long story short, my initial troubleshooting didn’t immediately lead me back to these missing permissions due to a number of other concurrent infrastructure changes which lead me astray. Additionally, when we tried to delete the Search Service Application to recreate it, the deletion failed after removing just one of the Search databases. Eventually we managed to re-provision the Service Application but the topology changes failed again, at which point we identified the missing %WINDIR%\Tasks permissions (again) and granting the missing permissions fixed these problems (almost).

In fact, we also needed to grant missing permissions on \Program Files\Microsoft Office Servers\14.0\Data\Office Server, but I believe that was a one-off related to the failed Search Service Application deletion earlier. One way or the other it doesn’t appear to be a core issue here. However, I should also mention that I suspect the Search Service Application deletion failed because of the missing %WINDIR%\Tasks permissions – although I’m basing this entirely on the fact that the ULS events above suggests that a similar process takes place for deletion, by virtue of the “(un)provisioning” job.

Round III

With Search back up and running, we moved on to other things, but eventually Search started acting up again. Unfortunately I’ve lost track of the visible failure, but the application logs were full of 6398 and 6482 errors (which typically indicate the unavailability of the service rather than the cause). I vaguely recall that we had items in the index but that new crawls were failing to run. At the time, I was most focused on Gatherer Access Denied messages on the Portal_Content Catalog.

Again, to abbreviate other misguided efforts related to on-going infrastructure work, we eventually found out that the permissions on %WINDIR%\Tasks were missing. Obviously, at this point the most reasonable explanation for the change was a Group Policy setting, so we reviewed the event logs in between the last known good crawl and the first crawl failure. I quickly spotted a Group Policy change message. I recommended that we review the Resultant Set of Policy on this server, just to be absolutely certain the Group Policy wasn’t applying permission changes in this location. The client assured me this was very unlikely, because they don’t have an overly restrictive culture, but it turned out this was the one and only file system permission change and it was applied to the Default Domain Security Policy. Presumably the previous Search failures occurred after reboots or some other event that would re-apply this group policy. And presumably all of this strange behaviour can be accounted for by these missing permissions, given that we know they were getting removed and we know that adding them back in fixed the problem.

Conficker

Later that night, curiosity got the better of me. I dug a bit deeper to see if I could identify anything that recommends these permission changes. I found Microsoft Support KB article KB962007, Virus alert about the Win32/Conficker worm. In this article, Microsoft recommends the following mitigation steps to prevent the virus from spreading:

Set the policy to remove write permissions to the %windir%\Tasks folder. This prevents the Conficker malware from creating the Scheduled Tasks that can reinfect the system.To do this, follow these steps:

1. In the same GPO that you created earlier, move to the following folder:

   Computer Configuration\Windows Settings\Security Settings\File System

2. Right-click File System, and then click Add File.
3. In the Add a file or folder dialog box, browse to the %windir%\Tasks folder. Make sure that Tasks is highlighted and listed in the Folder dialog box.
4. Click OK.
5. In the dialog box that opens, click to clear the check boxes for Full Control, Modify, and Write for both Administrators and System.
6. Click OK.
7. In the Add Object dialog box, click Replace existing permissions on all subkeys with inheritable permissions.
8. Click OK.

In effect, this Group Policy removes the special Read/Write permissions assigned to Authenticated Users on the %WINDIR%\Tasks folder by default. Note: it replaces all permissions with those defined in the Group Policy. I suppose the moral of this story is not to apply security settings like this to the Default Domain Security Policy. But fair play to my client for the security diligence in the first place.

Default %WINDIR%\Tasks permissions for Authenticated Users, without the group policy

This issue raises a couple of other questions. What is the best way to handle this for SharePoint servers, given that there are legitimate reasons harden this location? I suppose the best option would be to create another Group Policy for the SharePoint servers OU which will add the local WSS_WPG and WSS_ADMIN_WPG group permissions back on the %WINDIR%\Tasks folder. There will be other options, depending on how your domain/Group Policies are structured, but this illustrates an approach. It would be helpful to understand if the Search account should be added as well, but for now I’m going on what the installer/configuration wizard does rather than what TechNet fails to describe fully.

Next question: why isn’t this issue more common, given that the virus first emerged over two years ago? I suppose the group policy might not have been taken up by many organisations, but it’s more likely that there are further wrinkles I’ve not uncovered. I tried to replicate the problem in my single server + DC development environment, but frustratingly, everything worked fine after applying this group policy. I rebooted and confirmed the permission changes, ran a full crawl, ran a query and reviewed event logs, but all seemed fine. I even re-provisioned my Search Service Application and that succeeded. To be perfectly honest I’m not sure what to make of this. Perhaps this is only an issue once the search topology takes a specific shape? That feels like the most likely explanation. I hope to do more testing on this in future, but for now I wanted to identify a fix that worked for me and which aligns with the settings applied by the SharePoint installer/configuration wizard, should this problem arise for others. I’m not the first person to discover this problem. I think it’s actually been around since MOSS 2007, based on some forum posts, but I haven’t seen it described in relation to this Conficker protection, which hopefully helps make the Group Policy modelling decisions a bit less obscure.

More broadly, I’d be really curious to hear if anyone has information about the mismatch between TechNet and SharePoint default permissions on %WINDIR%\Tasks, and the further mismatch between the “Modify ACL” event, TechNet and the default settings. It may turn out that the WSS_WPG permissions are unnecessary or even undesirable, but given that SharePoint puts them there in the first place, I’m uncomfortable removing them until there’s better information to rely on.

A quick word about the DCOM Permissions

In my last post, I put off a discussion of the security implications of granting the Farm account DCOM Local Activation rights to the Windows Installer Service (in order to clear the DCOM 10016 event log errors). I was worried about this approach, since this DCOM Component opens up the Windows Installer, which represents a different type of security risk than say… IIS WAMREG. Following my last post, Spencer Harbar suggested that these worries were unfounded, or rather, that the risks are acceptable, since it’s only a risk if the Farm account gets compromised. He rightly pointed out that you’d be pretty stuffed at that point anyway. Fair enough. To this end, I’ll join him in not worrying about it.

How to fix it
If you want to clear the DCOM 10016 errors by granting these rights, you need to assign ownership of HKCR\AppId\{000C101C-0000-0000-C000-000000000046} to Administrators, then grant Local Administrators Full Control. Now you’ll be able to grant the DCOM Local Activation rights to the Farm Account on this same {000C101C-0000-0000-C000-000000000046} component.

Despite carrying a lighter weight on my shoulders, I think it might be helpful to review what came out of my testing, as the job may not be detecting everything we’d expect at face value. I’ve also poked a few more holes in the Support response, which was the whole reason I started working on this in the first place.

Testing the Job

In these tests, I’m wilfully trying to do stuff you would never want to do in any farm – just to find out what the job “knows” about. To this end, I’ve tried some pretty foolhardy things like:

• Manually updating DLLs in the GAC.
• Manually updating DLLs in the Program Files directories.
• Manually killing a Cumulative Update installation while it was half-way complete.
• Deleting DLLs from the GAC and the Program Files directories.
• Manually updating registry keys.

Are these the right tests? They certainly aren’t comprehensive. Suffice it to say I’m not the right person to comment on what the Windows Installer might be able to detect. In the process of researching this I’ve already become far more acquainted with Reflector and the Windows Installer than I ever hoped to be. I’ve even found out that there’s a Windows Installer blog and Windows Installer MVPs. Who knew? But are these changes the types of things that could cause disruption in a farm? Probably. And should we understand if the Manage Patch Status page in Central Admin accounts for problems like these? I think so. Thus, this imperfect testing by the wrong person.

Replacing DLLs

In the first two tests below, I copied DLLs out of an installed instance of the December Cumulative Update and replaced the installed June Cumulative Update versions of these DLLs in another machine with these newer copies. The DLLs I was looking at were for Microsoft Excel Services Components and Microsoft InfoPath Forms Services (this is how they are listed on the Manage Patch Status page).

Manually replacing a DLL in the GAC

When I manually deleted my June CU Microsoft.Office.Excel.Server DLL from the GAC using GACUtil (as you shouldn’t do), and replaced it with a newer version from the December CU, I broke my Excel Services Service Application. When I ran the Product Version Job timer job it failed to detect the change (the new version was never reflected in Manage Patch Status). Everything looked exactly as it normally would in the application event log, except for this message immediately after the normal 1015/1035 entries:

The Execute method of job definition Microsoft.SharePoint.Administration.SPProductVersionJobDefinition (ID 9bb9d31b-7c8b-4fd7-b52d-5fec40aa3607) threw an exception. More information is included below.

Failed to call GetTypes on assembly Microsoft.Office.Excel.Server.MossHost, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c. Method ‘IsEditEnabledForCurrentUser’ in type ‘Microsoft.Office.Excel.Server.MossHost.MossHost’ from assembly ‘Microsoft.Office.Excel.Server.MossHost, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c’ does not have an implementation.

This error is informative, and would probably help me track down the issue in due course, so the Product Version Job is earning its keep, but it’s unfortunate that this version change is not displayed in Manage Patch Status in any way. In short: this is a good reason to run the job but it’s also good to know this kind of problem won’t appear in Manage Patch Status.

Manually Replacing a DLL version in the Program Files directories

Next, I tried to manually replace DLLs in the Program Files directories with newer versions. I searched throughout the Hive and the C:\Program Files\Microsoft Office Servers\14.0\ directories for other versions of these files. I was working on the assumption that the version in the GAC would be in use (thanks to Chris O’Brien for this advice), but I wanted to see if the job would successfully spot changes in these Program Files locations, since this is what the Microsoft Support response suggested.

I found the same InfoPath DLL and a differently-named Excel Services DLL in these locations:

• C:\Program Files\Microsoft Office Servers\14.0\Bin\Microsoft.Office.InfoPath.Server.dll
• C:\Program Files\Microsoft Office Servers\14.0\Bin\xlsrv.dll

I ran the Product Version Job after deleting these files and rebooting. Again, the job failed to detect the changes.

What happens with added DCOM Local Activation rights?

If the farm account has DCOM Local Activation rights on the Windows Installer Service, it resolves the DCOM error event log clutter, but these rights don’t impact whether the job can detect these changed DLLs.

Killing an installation part-way through

Next, I rolled back to a stable state and ran the December Cumulative Update against a June Cumulative Update installation. At a random point during the installation I killed the installer (not the Products Configuration Wizard). While the installer was running I wasn’t able to monitor activity in ULS Viewer because SharePoint was being patched. However, I was looking at the dbo.ServerVersionInformation table in SQL Management Studio and I could see new rows with updated versions appearing as it progressed. The Cumulative Update installer was writing to the same table that the Product Version Job updates.

Running the Products Configuration Wizard after fixing the failed installation

Later, I fixed up my December CU installation and ran the Products Configuration Wizard. When it was running, I could see that something very similar to the Product Version Job was logged. The same informational events (1035) appeared successfully in the application event logs, without any DCOM errors or “Failed to Connect to Server” (1015) application event log warnings. Presumably this succeeds (with or without the DCOM rights) because the Setup account that’s running the wizard is a local admin and therefor already has the DCOM Local Activation rights. However, I’m not sure what’s gained by updating Manage Patch Status at this point, since the dbo.ServerVersionInformation table was already updated by the installer. I won’t dwell on that thought too much though, since there may be a very good reason for the update at this time.

For those who are interested in the workings of this update, it’s worth noting that the Products Configuration Wizard appears to use the Microsoft.SharePoint.Administration.SPServerProductInfo.UpdateProductInfoInDatabase(Guid serverGuid) method. It effectively calls the same thing as the Product Version Job timer job, if I’m reading all of this right. A fuller glimpse of the ULS logs looks like this:

Updating SPPersistedObject SPServer Name=SPSQL. Version: 120278 Ensure: False, HashCode: 2459215, Id: 20c667df-1bc3-486b-869c-a3ba40f83af5, Stack:
at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdate()
at Microsoft.SharePoint.Administration.SPServerProductInfo.UpdateProductInfoInDatabase(Guid serverGuid)
at Microsoft.SharePoint.PostSetupConfiguration.FinalizeTask.Run()
at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()
at System.Threading.ExecutionContext.runTryCode(Object userData)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()

It’s also worth noting that this log entry correlates with the MsiInstaller 1035 success events in the application event logs that I mentioned above.

What about deleting DLLs?

While investigating this, I ran all of this by my colleague Jalil Sear. He came up with an interesting idea: that I shouldn’t just update the DLLs, but I should try to delete them altogether. So I deleted Microsoft.Office.Excel.Server and Microsoft.Office.Infopath.Server from the registry and the GAC and reset IIS. I re-ran the Product Version Job and it completed normally, with and without DCOM Local Activation rights. Nothing was detected, although my entire Manage Service Applications page was annihilated. Again, we might have expected this to be reported in Manage Patch Status.

Summary of Test Results

• The Product Version Job reports “Success” in the Timer Job Status, regardless of all of these considerations. It may fail for other reasons, but all of these issues obtain when the job reports a successful status. In other words, the job reports “success” with or without DCOM rights.
  
• It’s not clear to what extent the Product Version Job can account for problems while the installer runs, because the installer already makes updates to the dbo.ServerVersionInformation table as it goes.
  
  • One might reasonably wonder what would happen to whatever was being updated while the installer failed. Obviously it’s hard to make broad statements about that when we don’t know at which precise point it failed, but in any case the remedial action will be to run the installer again - potentially after fixing something else. One way or the other, if you have this problem, I don’t see how the timer job is going to help because it’s unlikely it will be able to run against this server until the installation is fixed.
• It’s also not clear to what extent the Product Version Job can account for issues that occur while the Products Configuration Wizard is running – effectively for the same reasons as above. If you have a problem with that wizard, the remedial action will be to fix the problem and run the wizard again.
• Manage Patch Status doesn’t seem to account for other issues in the GAC or the Program Files directories, such as manual changes to DLLs. Presumably this is because these actions have been taken without using the Windows Installer Service.
  • Obviously, if you’re running an environment where these sorts of changes are routinely possible, then this job is a lesser concern than Change Management processes that might prevent these things from happening in the first place, but it’s worth knowing that the job did not detect these changes in my tests.
• It’s not clear in which cases the Product Version Job is useful for recording the difference between product versions on different servers, since the installer should have already updated the dbo.ServerVersionInformation table.
  
  • One example where the job might be useful is the case where a server is restored to a pre-upgrade state. However, it’s likely that this restore operation will prompt some other remedy, like reverting all of the other servers in the farm or upgrading this server again. So the usefulness feels limited to me. Still, this is probably sufficient reason to run the job absent any other considerations.
• The Manage Patch Status page is still useful for tracking differences across servers where the servers are legitimately running at different patch levels, although typically that’s not a state you’d want to run in for long.
  

Putting this information to use

I wouldn’t suggest reading this as the full story, since I only ran these against a single SQL/SharePoint box. At a minimum the Product Version Job can detect product version mismatches when a server is restored, and servers in long-term mismatched states. As a plus, it will throw an error in your application logs to let you know if there’s something wrong with the DLL that it expects in the GAC. Unfortunately, that isn’t reported to Manage Patch Status. In any case, as teams/farms increase in size this job becomes more useful for shared understanding.

At the end of this review, I think the important thing is to recognise the limits of the data in Manage Patch Status. It’s not going to be bullet-proof. For any actions taken with the Windows Installer, this data should be pretty reliable, since it’s updated during install, with the Products Configuration Wizard and with the Product Version Job. For anything else - who knows? It doesn’t appear to have been designed for that, and I have no idea what a SharePoint timer job would look like that could offer these kinds of assurances. Presumably it would have to be a management agent of some sort. At that point you’re in to Configuration or Operations Management territory and we already have different tools for that. Come to think of it, if you really want to know, “the install state of the machine“, that’s probably what you’re really looking for. But if you want to know the current versions of successfully-installed SharePoint Products on all servers in your farm, then Manage Patch Status should be accurate in most cases, because of the Product Version Job.

This is a fairly lengthy consideration, but I think it’s necessary to cover these details because the information in the Managed Patch Status (AKA Check Product and Patch Installation Status) page in Central Administration may not be revealing what we’d reasonably infer.

In this post and the posts to follow, I’ll cover a few things:

• Why I think granting Local Activation rights to the Windows Installer Service puts a dent in the least-privileged model.
• What this DCOM error means to the reliability of data displayed in the new Manage Patch Status page in SharePoint 2010 Central Administration.
• What the job does and doesn’t do, with or without rights to launch the Windows Installer Service.
• Considerations for disabling the Product Version Job timer job.

The Problems

I believe most people will come to this problem in the way that I have, which I’ve seen repeated on many TechNet fora since then. People want to know why they are getting inundated with approximately 100 DCOM 10016 System event log errors and twice that many MsiInstaller Application event log warnings and informational events nightly, at around 00:52. The exact number of messages will vary based on the SharePoint products installed in the farm, including related products such as Project Server, Office Web Apps, FAST Search, etc. For a more detailed review of these events and how they can be identified, please refer to my original post.

Additionally, we have a nightly timer job which seems to be failing, per these DCOM errors. The job itself claims to check, “the install state of the machine and puts that data into the database”. This is rather vague. As of August this is what I understood:

• The timer job appeared to fail to use the Windows Installer Service to perform a check of installed SharePoint products.
• I didn’t know anything about how that check happened or how the data was used afterwards.
• I didn’t know if the event log messages were ephemeral (annoying only because they generate clutter), as they are for the IIS WAMREG DCOM 10016 errors.
• I felt it would be bad to grant rights to launch the Windows Installer Service to the farm account in an otherwise-least-privileged configuration (where the Farm account does not already have local administration rights).

In this post I want to dwell on the inner workings of the  job itself, and then come back to the implications for our event logs, permissions and job scheduling.

Inside the Job

In order to find out how the job works, I had to crack it open in .NET Reflector and SQL Management Studio. I need to disclaim this post, because I’m not a developer, and to be perfectly honest I’m in a bit over my head with Reflector, but I was prompted to investigate in this way based on the apparent misinformation in one of the TechNet threads I mentioned above. Geoff Belair went to considerable lengths to work through this topic with Microsoft support, but from what I can tell, there are a number of mistakes in the answer he received. It suggests the wrong database gets updated and is a fairly inaccurate description of what this job does, by my reading of the following clues.

(In)validating the Microsoft Support Explanation

It’s unfair to take a Microsoft Support e-mail which has been re-posted on the web as authoritative, but this was the closest thing to official information I’ve found, other than the brief words about this job on TechNet and MSDN. The key bit of that reply that I wanted to immediately verify was this:

The Timer Job “Product Version Job” runs every night at 12:52 A.M and analyze which are the dlls are updated, once it get the information then it’s put the updated version data on to Content Database “dbo.version” table.

So I took a look at the dbo.Versions table in the Central Admin Configuration database (never do this in production, of course).

What caught my eye was that there was no product information in this table whatsoever. I knew that the job was checking for the state of individual products based on the MsiInstaller informational events in the Application logs. So I poked around a little more and found what I was expecting in the dbo.ServerVersionInformation table:

Having looked at this data, I realised it was pretty familiar. I went back to Central Administration, looked in the Upgrade and Migration section and clicked on Check Product and Patch Installation Status, which took me to this Manage Patch Status page. The key thing to note is that the version numbers and the Patch Status columns match the data on the page below precisely. I’ve actually manually updated that data just to give it a sneaky check, and this page is definitely pulling it in from that source. You’d never do this on a real system, however. I wouldn’t even do it without having a recent snapshot for my development environment.

At this point I was pretty confident the timer job was trying to update this table, but I wanted to get a bit better assurance before testing the job in anger. I also wanted to understand how the Windows Installer Service gets involved, as this activity seems to take place outside the ULS logs.

Analysing the job in .NET Reflector

Cracking open ULS Viewer while running the timer job, you immediately see the fourth event in my screenshot below. Job-admin-product-version calls SPProductVersionJobDefinition.

This is where I opened Reflector. I started with Microsoft.SharePoint.dll and drilled down to Microsoft.SharePoint.Administration.SPProductVersionJobDefinition, which executes SPServerProductInfo.UpdateProductInfoInDatabase(Server.Local.Id);

SPServerProductInfo calls the GetMsiData method, which works with a number of SPMsi methods (SPMsi.GetPropertyUsingProductCode, SPMsi.MsiEnumPatchesEx, SPMsi.MsiGetPatchInfoEx, SPMsi.SPMsiSafeHandle, SPMsi.MsiOpenDatabase, SPMsi.MsiDatabaseQuery). Further down, SPProductVersionRow is clearly collecting the same data as the columns of the SQL dbo.ServerVersionInfromation table I examined earlier. If interested in these workings, I’d recommend perusing it with Reflector at a more leisurely pace than this.

Note: all of the SPMsi.Msi* methods are using msi.dll, which is the Windows Installer.

The Product Version Job’s use of the Windows Installer

From here, I could explore the workings of the Windows Installer in finer detail – but for the purposes of our SharePoint knowledge, all that’s really important to know is that the timer job is using the Windows Installer’s own methods to query the installed product versions on the servers. As I understand it, the Windows Installer typically stores this data in the registry, at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\. That’s an oversimplification, but stick with me for now.

We can see a number of SharePoint products in this location. They are all keys beginning with “90140000″. Taking a look at the data in these keys, it’s pretty clear that it aligns with the data that’s written to SQL’s dbo.ServerVersionInformation table (down to the registry key value in the “Patchable Unit” column). Additionally, these are all the same products that are identified in our Application event log messages. You can even see the patched products have a longer key, with a suffix that looks something like “_Office14.OSERVER_{48017E90-141F-4948-A576-F4B9B6284B70}”.

Perhaps most importantly, the ProductVersion Property of the Windows Installer is what defines the four “version” values (including “DisplayVersion”) of the Uninstall keys above. This is the key information that the Product Version Job is after, and the name of this timer job feels like an even better fit in this context.

While unravelling the job in this way has given me a fair amount of confidence about how SharePoint retrieves this information, there are still a number of issues to consider. For starters, I suspect people look at Manage Patch Status data and feel pretty confident about that representation of the installation state of their servers. Being a fairly skeptical type, I suspected that the Windows Installer’s “record keeping” would be good up to a point, but no further, so I put on my demolition hat and started breaking stuff, in an effort to place that point. In my next post I’ll review those test results, then consider the implications for DCOM rights to the Windows Installer Service and the timer job scheduling options.

• SharePoint Development Productivity and Virtualisation Technologies
• SharePoint 2010 Development Environment Performance Tests

I’ve said my preamble in those posts, so I’ll cut to the chase here.

High-Level Summary of Findings

• Disk performance and bus speed did not prove to be significant factors in these results (except for virtual machine start-up times). Obviously there are fundamental differences about SSD (yet untested) that may skew this picture, but I will be surprised to see big differences. If we’ve got these tests right, and they are actually representative of the tasks that slow down development, then we would expect to see wider variance across bus or disk speeds. We don’t.
  • This assumes the disk is relatively uncontended. Virtual machine performance degrades in every type of test while large file operations are running concurrently on the same disk. This could be copying an ISO, importing or exporting a virtual machine or any other sustained large file operations.
    • At a minimum, this is certainly an argument for running VMs on their own spindle, whether it’s over USB, eSATA or SATA. This may be an area where SSD shines.
  • These disk performance figures can be found towards the bottom of this post. Desktop performance was nearly identical running on USB2 at 5400 RPM versus a RAID0 stripe or a RAID1 array on 7200 RPM disks. Laptop performance was also nearly identical over USB2 5400 RPM versus eSATA 7200 RPM.
• Hyper-V performance has been poor on all laptops with i-Series CPUs. This is more pronounced in some areas than others. Our three-year-old model with a Core 2 Duo actually outperforms the new i7 in some cases. When these results are added to known driver issues with Hyper-V on many newer laptop GPUs, we’re looking at a configuration that’s unfit for SharePoint 2010 development.
• VMWare Workstation outperforms Hyper-V on laptops by significant margins in most areas. The exceptions to this are start-up time and performance during the first 10-30 minutes of use (I believe VMWare is ballooning during this time). After that, VMWare Workstation is faster than Hyper-V in every type of test.
  • As a long-time advocate of Hyper-V despite usability deficiencies, I was probably more surprised by the significance of these differences than anyone. I wrongly assumed that Type-I hypervisors would outperform Type-II in nearly every way. While that may hold true on server class hardware, it doesn’t hold true here. I’m a convert.
• While less pronounced, these same findings hold true on the desktop.
  • Desktop performance is very quick on VMWare Workstation, considerably out-performing even Amazon EC2.
  • We can realise significant productivity gains by moving all users who are primarily office-based to a desktop + VMWare Workstation configuration from laptop + Hyper-V, at a fairly small cost (probably half the cost of EC2 over three years – see my recent posts on EC2 for more information).
  • Desktop performance on Hyper-V, while notably slower than VMWare Workstation, is generally faster than VMWare Workstation on the i7 laptop.
• Laptop performance is significantly improved on our current model with VMWare Workstation. These improvements are also realised on the newer model laptop, but the performance delta between the two physical systems is not so significant that it’s compelling to move to a low speed i7 from a reasonable speed Core 2 Duo.
  • The total times for the “End-to-end site creation to debugging tests” were two and a half minutes faster with VMWare Workstation compared to Hyper-V on the Dell XPS M1330. Moving from Hyper-V to VMWare Workstation for laptop users is now an obvious choice.
  • The benefit of spending on i7 processors is in doubt. We are seeing very minor performance penalties when adding more than two CPUs in both VMWare Workstation and Hyper-V for most tests. There were also very minor improvements for some tasks, but on the whole there does not appear to be a measurable benefit. This might vary if the host OS is doing a great deal with the CPU, but that is liable to cause other contention issues than just in the CPU (on a laptop).
  • The only tasks that appeared to use all 8 cores in a SharePoint VM were:
    • Retract/Deploy of a solution (but only very briefly)
    • Create web app, or Create site collection (but at low percentages)
    • Rebuild with Code Analysis (but not fully)
  • We will be running future tests on i5 processors at higher clock speeds to see how these models perform relative to the 1.6 GHz i7.
• The User Profile Service Connection doubles first page load times after an IISRESET in all test cases. I consider this a full validation of these preliminary findings.

Snapshot of key data

The Data

How to read the data:

• Hardware: the physical laptop or desktop model (or Amazon’s EC2)
• Virtualisation: “Hyper-V” is short-hand for the Hyper-V role in Windows Server 2008 R2. “VMWare 7.1.2″ is short-hand for VMWare Workstation.
• #CPU: the number of physical CPU presented to the guest operating systems. Multiple logical cores were only tested in the 4×2 results below.
• Disk: the physical disk configuration where the virtual hard drives are running.
• RAM: the amount of RAM running inside the SharePoint Server 2010 VM. The Amazon EC2 instances were “large instances” but the domain controller was running locally.
• Test: The tests have been described in more detail in my last post.
• Result 1, 2, 3: Each test was carried out three times. The far-right column, Average Result, is an average of the three.
• The Two “Average Load…” rows are an average per-result of the three rows above them. These are tests built on SharePoint 2010 default site templates, which anyone should be able to replicate.
• The “Total create to debug time” row is a sum of the five rows above it.
• All results are in seconds. In cell G21 below, 524 seconds = 9 minutes and 2 seconds.
• For more information on the tests and the testing methodology, see my last post.

Hyper-V versus VMWare tests, all other things being equal

Dell XPS M1330, running Hyper-V

Dell Studio XPS 1645 laptop, running Hyper-V

ASUS V7-P7H55E desktop, running Hyper-V
Note: these Hyper-V tests were accidentally carried out while the VM was running on a RAID 0 stripe rather than on the System disk, so this is not apples and apples, but later disk tests on VMWare Workstation indicated that this shouldn’t make much of a difference, so I’ve left these results in, with this comment.

Dell XPS M1330, running VMWare Workstation

Dell Studio XPS 1645 laptop, running VMWare Workstation

ASUS V7-P7H55E desktop, running VMWare Workstation

VMWare Workstation i7 tests with 4 or 8 cores

Dell Studio XPS 1645 laptop, running VMWare Workstation with 4 CPU

ASUS V7-P7H55E desktop, running VMWare Workstation with 4 CPU

Dell Studio XPS 1645 laptop, running VMWare Workstation with 4 CPU, 2 Cores Each

ASUS V7-P7H55E desktop, running VMWare Workstation with 4 CPU, 2 Cores Each

Amazon EC2 Results

Notes:

• Times were much slower one day than others. This hasn’t been measured over time, but it’s worth keeping in mind. Other EC2 users reported similar problems on the same day.
• Also note: a couple of rows of test data (245 and 248) have been accidentally deleted, but the results were not unexpected in any way.
• Row 263 has no data because measuring time to desktop with EC2 would be too imprecise. It would normally be available within five minutes from start, for reference.

Disk Tests on VMWare Workstation with two cores

The format of these tests change slightly, as I am grouping all disk permutations for the Dell Studio XPS 1645 together, then moving on to the ASUS V7-P7H55E desktop. I grouped them this way because the tests were fundamentally different for laptops and desktops. I did not get the time to repeat the laptop tests on the Dell XPS M1330.

Dell Studio XPS 1645 laptop with VM running on 5400 RPM USB2

Dell Studio XPS 1645 laptop with VM running on 7200 RPM eSATA

ASUS V7-P7H55E desktop with VM running on 5400 RPM USB2

ASUS V7-P7H55E desktop with VM running on a 2nd set of RAID 0 spindles

ASUS V7-P7H55E desktop with VM running on a 2nd set of RAID 1 spindles

…and with that, I’ll let you draw your own conclusions. Should anyone want to contribute supplementary test data in the comments here, or carry out further tests (perhaps with SSD), I would love to see the results. As I mentioned in the last post, there’s still more testing to do.

Update 08 June 2011:SharePoint 2010 Development Environment Performance: SSD, i5 vs. i7, WEI and Sandy Bridge

The Tests

The 21 tests that we settled on were the result of discussions with a number of the core developers, consultants and architects at Content and Code, plus a few tests that I threw in to confirm/disconfirm some of my suppositions, such as the impact of the User Profile Service Connection on first page load time. All 21 tests were run three times for each permutation of hardware candidate and virtualisation technology. We also tested on Amazon EC2. I will discuss the testing process in more detail in a moment.

These tests have been selected for a few reasons:

• They are tests that anyone can run, including Visual-Studio-allergic types like myself.
• They re-enact real-world productivity loss. All tests needed to be significant on our current system or they were thrown out.
• They needed to account for tasks that impact non-developers as well as people that have their head down in code 40 hours/week.
• They needed to be examples of tests that would stress systems in different ways.

First page load tests
These tests were designed to examine what, if any impact different sets of features, functionality and structure might have on first page load times after the application pool is recycled or IIS is reset (while gathering a large set of data to make comparisons across systems). I also wanted to fully validate my preliminary findings about the User Profile Service Connection.

I ran these tests against NTLM-authenticated web applications with the following root site collections:

• Central Administration
• Blank Site
• MySite
• Blank Site, with no User Profile Service Connection
• The Content and Code website solution (structure, without content)
• A custom intranet solution (structure, without content)

All of these first page load tests were repeated for application pool recycles and IIS resets.

End-to-end site creation to debugging tests
I hope these tests are fairly self-explanatory. I used the Content and Code website solution because it’s a public site that people can examine if they want to understand more about the structure of the solution and the scope of customisation tested here.

1. Create new NTLM-authenticated web application from the GUI
2. Create new Publishing Portal Site Collection from the GUI, at the root of the new web application
3. Deploy Content and Code website solution from Visual Studio
4. Delete the publishing site collection (this was a necessary step, but not a test that I timed)
5. Create Content and Code website (structure, without content) from the GUI
6. Debug Content and Code website solution in Visual Studio

Core development tests
These tests were added to account for pure development activity for large projects with lots of dependencies. We turned Code Analysis on for the first test because this is a feature that’s very useful but taxes systems pretty heavily. The code deployment times were all fairly small relative to other tests here, but we need to keep in mind that this could be repeated literally hundreds of times per-day. Note: full deployment is accounted for above in the end-to-end test.

• Rebuild Large Project w/Code Analysis
• Deploy Large Project to GAC/BIN

Disk/IO tests
These tests were thrown in because they have an impact on productivity even if they aren’t particularly routine. For the first test I measured the time from turning on the VM until the desktop rendered after logging on. The second test doesn’t really meet the “real world” criteria I name above, but it is a task that can be a productivity barrier in some cases.

• Time to desktop
• Run full crawl (three web apps, no content)

The Testing

The testing process was entirely subject to personal fallibility, as I carried these tests out myself using fairly imprecise methods like a browser-based stopwatch running on my host system (I made sure not to time things inside the guest, where time can slip occasionally). I also went to great lengths to carry out these tests when the systems were performing optimally; I would run through all of the tests once before recording the first set of results. I felt this approach was the best way to discount random variance. The test results were largely very consistent, so I believe these efforts paid off. Obviously the down-side to testing in this manner is that real work is not carried out in a vacuum, but I don’t see any other way to come up with repeatable tests aside from measures like these. It’s what works for science, after all.

The Virtualisation Technologies

As I mentioned in my last post, I chose to limit the virtualisation technologies to a single technology from each of the types I described. I had to postpone testing against “local systems” due to time pressures. It was the option that fell off because we are unlikely to ditch virtualisation any time soon. It works well for us.

To reiterate here, the candidate technologies were VMWare Workstation 7.1, the Hyper-V role in Windows Server 2008 R2 and Amazon’ s EC2 IaaS offering (a Red Hat implementation of the Xen hypervisor). Again, there’s background for all of this in my last post.

What About the Server Room?

One thing I haven’t discussed in any detail so far is VDI or Remote Desktop services. I briefly touched on shared development environments, but I’ve not talked about hosted, individualised development environments. The reason we ruled this out is cost. While this would probably be the best-performing option, all other things being equal, the costs associated with providing this level of performance in the server room would be pretty enormous. For our purposes we might have exceeded power, cooling and weight limitations before we considered the costs of new blade centres and SANs. These costs would probably be even greater in the datacentre. In short, the same criticism applies to individualised hosted development environments as to shared environments: redundancy and resilience at this level is overkill given the associated costs. The data is not critical and anything that needs to be backed up can be stored elsewhere (like TFS).

Basically, people opt for VDI or Remote Desktop services because a mass of underutilised desktop systems can be heavily consolidated. These systems are not underutilised.

The Hardware Candidates

Dell XPS M1330
This is our current laptop model, upgraded with a 320GB 7200 RPM local hard drive and 8GB RAM. One of the serious options we’re considering is a laptop refresh, due to the age and fail rate of the graphics cards and motherboards on these models.

Dell Studio XPS 1645
This was the least expensive decent i7 laptop I could find for testing purposes, and a leading candidate as a replacement laptop. With an £833 (ex-VAT) starting price it could be bumped up to 8GB RAM for a little over £100 more via Crucial. It’s a very heavy laptop and the glossy shell does it no favours, picking up fingerprints within seconds of use. However, it comes wth a 1.6 GHz i7 processor, 500GB 7200 RPM disk standard, eSATA port and HDMI. No USB3. Basically, nothing here was an absolute deal-breaker for us if performance was good.

ASUS V6-P7H55E
This is a barebones system with the following configuration/cost (as priced at scan.co.uk):

• ASUS V6-P7H55E barebones System = £121.67
• Intel i7 870 (8M Cache, 2.93 GHz) = £217.57
• 4GB Corsair XMS3 DDR3 PC3-10666 (1333) Dual Channel – 4x£56.59 = £226.36
• 1TB Seagate Barracuda SATA 3Gb/s, 7200rpm, 32MB Cache, 8.5 ms, NCQ – 3x£41.94 = £125.82
• Adaptec 1220SA PCI-E RAID Card = £46.40
• ASUS 512MB GeForce G 210 DDR2 NVIDIA Graphics Card = £27.71
• Total = £768.58 (VAT-inclusive)

This system is configured with three internal 1TB hard drives and 16GB RAM. We needed to purchase the RAID card because the motherboard does not have an on-board RAID controller. The graphics card was necessary because there are no integrated graphics on desktop i7 processors (although there are for some i3 and i5 models). The disk configuration was variable, as this was one of the test scenarios. The assumption going in was that two disks would be configured in a RAID 0 stripe or a RAID 1 array, depending on performance outcomes. We would only stripe the disks if there was an obvious, significant performance gain. The third disk would be attached to the on-board SATA controller. I will discuss the recommended configuration in more detail later. Also note: the graphics card supports two monitors across any two of the three outputs, but not three concurrently. Finally, the ASUS V7-P7H55E is nearly identical in every respect. We went with the V6 based on availability.

Other laptop models
During preliminary testing we looked at the Lenovo W510, the Dell Precision 6500 and the Alienware M17x among others. All of these models were candidates that we never ruled out, but we didn’t have sufficient time with them to run the entire set of tests. However, these models had a reasonably similar configuration to the Dell Studio XPS 1645 and the Hyper-V tests we ran on these systems yielded similar results to our test model.

Other desktop models
Obviously a barebones system won’t appeal to everyone as a business solution, and it took me some time to persuade myself that it might be suitable for these environments. It wasn’t until I actually priced up this model and compared it to the comparable Dell T1500 (+~£600) and HP Z200 (slower than either model, and pricier) that I considered how it might work for us more seriously.

What am I examining, and not examining?

We have an old laptop, a new laptop, a new desktop and the cloud. Excepting the cloud (which is fixed), we’re permuting each of these hardware options with VMWare Workstation and Hyper-V test results. We’re then adding tests to examine the impact of spindle/bus speeds and the impact of adding/removing cores to these VMs. Ultimately, I wanted to quantify the productivity impacts of a change to our hardware and/or virtualisation technology as opposed to a change within our virtualisation technology, insofar as these tests could be decoupled.

I am not examining every virtualisation solution nor every hardware permutation but I do try to account for a number of these variables with these tests. I would love it if people carried out similar tests on their environments to help build knowledge in an area that’s hugely uninspected today. These are some of the other tests that I hope to revisit next year:

• The impact of application pooling on first page load times. Preliminary tests suggested there might be a small impact, but nowhere near as significant as the User Profile Service Connection. This warrants further inspection.
• The performance of “local systems” on this same hardware. As I mention above, these tests had to be de-prioritised, but I feel it would be worth identifying if there are any of these development-specific tasks where some, or all virtual technologies suffer.
• While I am running tests against a number of disk buses and configurations, I did not get the opportunity to test SSD performance. Obviously a lot of people will want to know the impact of SSD on these timings, but unfortunately I won’t have an opportunity to inspect that until early next year at the earliest.
• In some cases we work with deep snapshot trees. I want to gain an understanding of how differencing across ten or more files impacts performance for these tasks.
• Compare performance of a higher-clocked i5 to a lower-clocked i7 at a similar price range and potentially explore over-clocking options.
• Compare slower memory on an otherwise-identical system.
• Run VirtualBox tests on an otherwise-identical system.
• Assess the impact of virtualisation optimisations.

Obviously these tests say nothing about the usability of the system, power costs, mobility and more. For the purposes of this post I’m only concerned with outlining how I tested system performance for these real world tasks. In the next post, at long last, I will share the results.

For clarity, I will quickly state the problem as I see it. SharePoint 2010 system requirements and practitioner mobility requirements are inherently at odds. What guidance exists for this unique problem space tends to regurgitate preferences/allegiances rather than comparing technologies and ratifying assumptions with real-world tests. At best, you get system performance indices for a single laptop model, but these results may vary when any hardware component is changed.

How can virtualisation improve system performance?

It doesn’t. People look to virtualisation to solve other problems. However, SharePoint 2010 performs differently in different virtualisation technologies, and the margins of these differences vary by hardware configuration. By all means, the advantages of virtualisation often make it a desirable choice, but these performance characteristics need to be accounted for, lest system performance losses negate the productivity improvements that virtualisation can introduce.

Why virtualise?

There are a number of advantages to virtual systems over physical systems. Many of these benefits can also be obtained with sufficiently mature systems management technologies and physical systems, but these benefits are often easier, quicker or less costly to implement through virtualisation. Some of the benefits include:

• Provisioning times for new SharePoint environments.
• Standardisation through cloned, network-isolated virtual machines.
• Account for volatility with snapshots.
• Standard builds per-project, to share with team members, reducing project initiation costs.
• Virtual appliances produced by Microsoft and third parties, such as the Information Worker Demo VM.
• Reduced hardware rebuilds by removing development tools and SharePoint from the host.

This list is by no means comprehensive. As I say, many of these benefits can be realised with scripting and/or management tools. This list is only intended to illustrate why it’s a powerful design option.

An overview of virtualisation and related technologies

Some example technologies by type:

• Type I Hypervisors
  • VMWare ESXi
  • Hyper-V
• Type II Hypervisors
  • Oracle VirtualBox
  • VMWare Workstation
• Infrastructure as a Service (IaaS)
  • Amazon EC2
  • Azure VM Role (forthcoming)
• Local Systems
  • Native Boot Windows 7 (virtual hard disk)
  • Citrix XenDesktop (VDI)

Note: Virtual PC was not included because it doesn’t support 64-bit guest operating systems. SharePoint 2010 only runs on 64-bit systems.

Some of the alleged benefits of these approaches:

• Type I Hypervisors
  • Better performance**
  • Good management options/tools
• Type II Hypervisors
  • Host Operating System
  • Easy to use
• Infrastructure as a Service (IaaS)
  • Pay as you go
  • Scalability
• Local Systems
  • Good performance
  • Simple to use

Some of the alleged drawbacks of these approaches:

• Type I Hypervisors
  • No Host Operating System***
  • Driver issues*
  • Complicated
• Type II Hypervisors
  • Historically poor performance**
  • Historically, less manageable (snapshots, import/export, etc)
• Infrastructure as a Service (IaaS)
  • Requires stable connectivity
  • Complicated
  • Pay-As-You-Go requires diligence
• Local Systems
  • Easy to damage
  • Slow to rebuild

*Hyper-V has driver issues on some newer laptops. These are most noticeable with graphics, although I have seen audio driver problems as well. Some of these driver issues may be fixed or alleviated in the SP1 Beta/RC for Windows Server 2008 R2.

**This performance bias is one of the things I will be examining in more detail in later posts.

***This is only “sort of” true for Hyper-V, which invokes a “parent partition”. This is a special type of virtual machine that fulfils a similar role to a host operating system, and is often referred to as such.

Why are “Local Systems” included?

I’ve lumped these in for two reasons. 1) They share some characteristics with the other virtualisation technologies, like running from virtual hard drives. 2) By virtue of being local systems, they fundamentally negate some of the benefits that are obtained through virtualisation. Cloning these machines is not an option if SharePoint is installed and configured. It will be necessary to invest in scripting environment provision in order to retain those productivity benefits. It happens that many people choose to take this scripting approach, but it’s worth pointing out that network isolation and cloning can achieve similar results through virtualisation, and this does not obtain with Local Systems.

What about shared, hosted development environments?

In this scenario I’m thinking of hosted development farms, where some or all members of a team use a single environment. Based on my subjective reading of the community, this option seems to be fading away. I think there are three reasons why.

1. Cost. Running development environments on proper infrastructure is expensive. Most components have been made redundant, the storage will be expensive if it performs well, the power/cooling costs are considerably more expensive than for laptops/desktops and you will need to pay people to manage the systems. Even when these costs are split across multiple developers, it’s still expensive unless resources are overcommitted, which negates productivity gains. It also tends to be more expensive to provision new environments and this process can often be an obstacle to business agility. In a nutshell, these are protections that are unnecessary for development environments. Redundancy and resilience at this level is overkill given the associated costs. The most important assets, such as code, standard images and project-specific builds can be protected separately.
2. Hive pollution. If these farms will support multiple projects, as they often do per the previous comments about provisioning, then these systems will inherently differ from the test/stage/UAT/production systems they should resemble. Core files in the hive can be altered from project-to-project, resulting in unexpected behaviour when moving code between these environments. This can seriously complicate troubleshooting and should be avoided.
3. Mobility. These farms aren’t terribly useful to people who are travelling or who are working on-site with restricted outbound connectivity.

All of this said, there are times when project-specific requirements may make shared farms a good option. It may be sensible to take another look for:

• Integration projects.
• Developing with large amounts of data.
• Projects with heavy infrastructure requirements, such as FAST.
  • Perhaps individual development environments can consume a shared FAST Service Application?

Generally speaking, I believe these resources should be provided only in these niche cases.

How is this different from IaaS?

The main differences are costs and capital. Cloud-based infrastructure services are fundamentally just virtualised hosting on an enormous scale. This scale lowers costs to a point where it may be affordable to deploy individual machines per-developer. Although in my analyses I found that IaaS would be more expensive than desktop workstations over three years, this still may be compelling when cash flow issues preclude significant one-time investment or credit flows are restricted. IaaS should also be kept in mind when specific projects require significant provisioning or investment for a short term, for instance testing in a large farm.

While providing a single cloud-based VM per-user solves the first two issues with shared development environments, mobility is still an issue. In many places, stable mobile broadband is flaky at best. Additionally, there are key architectural differences that need to be accounted for when working in the cloud, and on a Pay-As-You-Go basis. I address all of this in my series on SharePoint 2010 Infrastructure for Amazon EC2.

Which approach is best?

This is a high-level overview of the design constraints that limited my choices, before I plunged into a concrete performance review of the remaining technologies.

Local Systems
In my view, Local Systems are only a better choice if the supporting IT systems and processes are very mature and the performance benefits are clear and significant. For most development scenarios, that has yet to be proven. I’ve postponed this virtual to physical performance comparison for now, as the other benefits of virtualisation have ruled this approach out for us, but I hope to revisit it in the new year.

IaaS
IaaS has two key planning considerations. The first is fairly obvious. Outbound RDP Connectivity needs to be open whenever the systems are needed. I encourage people to consider this in some detail and pilot with many types of users before diving in. The second consideration is Pay-As-You-Go. While cloud providers often have an always-on option, it’s usually pretty pricey. The alternative is to find a mechanism to limit compute usage to when it is truly being used, without introducing usability problems. Management tools or scripting may be able to answer these problems, but no one should enter in to this process thinking it will be easy. This is not an easy option. For a more detailed consideration of these issues, refer to my series on EC2.

Type II Hypervisors
VMWare Workstation is the most mature desktop virtualisation product on the market, although in recent years VirtualBox has been gaining share. Choosing between these technologies for my tests was never going to be easy, but I reduced it to a few factors:

• I’ve never met a VirtualBox user that would complain about using VMWare but I can’t say that proposition is reversible. There are a lot of SharePoint practitioners with a strong preference for VMWare.
• VMWare Workstation has native interoperability with other VMWare assets. While VirtualBox supports the VMDK file format, it’s not quite the same thing.
• Both products are fairly inexpensive in the grand scheme of things.
• I had stability issues with VirtualBox circa version 3.14 that left a bad taste in my mouth.

Perhaps most importantly, I felt that the performance comparison of VMWare Workstation to Hyper-V would be the most valuable decision-making information.

Type I Hypervisors
Most Type I Hypervisors would not be suitable for desktop virtualisation because they don’t have a host operating system. While it would be possible to boot a guest OS and remote in to other Virtual Machines over internal networks, this is a complicated approach and the networking requirements would be enough to put off most developers. However, as mentioned above, Hyper-V is a notable pseudo-exception to this with its parent partition.

We’ve been using the Hyper-V role in Windows Server 2008 R2 for development for a little over a year now. While we have successfully capitalised on many of the productivity benefits of virtualisation through this approach, there are a few issues that have never been entirely satisfactory:

• Despite having the host OS, using Hyper-V is still complicated for non-Systems people – particularly the networking.
  • Work-around solutions for Wireless networking are fiddly.
  • Lack of self-contained NAT requires the use of Internet Connection Sharing in order to achieve network isolation, which some users struggle with.
• Lack of Sleep/Hibernate is painful for many users.
• Graphics performance is poor – particularly with large PowerPoint/Visio files, large images and video.
• Audio can also suffer during large file operations.
• Hyper-V is not ready for laptop power schemes.

Despite these niggles, we’ve continued to use Hyper-V while waiting for the forthcoming graphics/memory improvements in Windows Server 2008 R2 SP1. I would class these usability problems as significant inconveniences that sometimes manifest themselves in lost productivity – particularly with new users learning our approach.

New Problems in SharePoint 2010

Since we properly immersed ourselves in SharePoint 2010 development, negative reports about performance started to roll in. These proved hard to validate until a few months ago when my colleagues showed me first page load times after an IISRESET in excess of one minute. This was concrete and repeatable. The problem was more severe on some systems than others, but it was clearly a problem.

The performance tests I’ve been conducting have been an effort to pick apart these results in Hyper-V. Was this new in SharePoint 2010 or did it amplify something that was minor before? Do we get the same problems on different virtualisation technologies, in the cloud or is this a symptom of virtualisation itself? In my next post I’ll discuss the environments, the tests and the testing process.

Other posts in this series

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis
• Amazon VPC and VM Import Updates

When would AWS be compelling, despite the complexity?

I’ve covered most of the design topics that I feel are relevant to SharePoint 2010 on EC2 now, so it’s time to talk about why we would use it, despite the obvious complexity that it introduces. The potential benefits included:

• Scalability. This is pretty hard to question. AWS definitely scales.
• Cash flow: The On-Demand services are Pay-As-You-Go, so this clearly helps when cash is tight.
• Infrastructure costs/support: This needs to be validated. See the AWS Premium Support Pricing page for more information about the cost of platform support.
• Performance: I will be diving much deeper in to performance over the next week or two and will be analysing EC2 alongside laptops and desktops. For now I will say that it performs well, but it isn’t the best-performing solution that we reviewed. Subjectively I would say that I don’t think most developers would consider a large instance to be slow.
• Availability anywhere (with an outbound RDP connection): Obviously the down side here is that this connection isn’t always available or reliable everywhere, for instance on a train.
• Special scenarios: Some examples I can think of here would include testing for large farms and office moves. I shan’t delve in to the scenarios, but there are sure to be others.
• Cost: This needs to be validated, and I will share an example analysis below.

Actual invoice data

This screen shot of an Amazon invoice (tidied up in Excel a bit) is the real invoice I received for my testing time. I’ve included it here because I think it illustrates the impact of instance usage time on total costs really well. It’s by far the largest cost at ~90% of the bill for this testing time, and that included a couple of weeks when I wasn’t using the instances. During that “down time” I was still billed for storage use and Elastic IP address disuse. Keep that in mind, as you will continue to accrue charges even if you shut down your machines.

Example of costs over three years

I projected charges based on these figures over three years for a large number of users. There were two main objectives for these calculations:

1. Gain an understanding of the impact of on-demand usage compared to reserved instance costs.
2. Assess these costs relative to hardware costs over an average lifetime of three years.

This analysis was only intended to indicate ballpark costs and some of the figures are nothing more than educated guesses, but I think they should serve their purpose as indications. This analysis didn’t factor in costs for Amazon’s Cloud Watch (monitoring and reporting), Amazon Support, licenses (other than Windows) and probably some other factors I overlooked, but I’m publishing it here as it might be useful for other high-level assessments. But obviously everyone should work this out for their own usage patterns and obviously, all price information is subject to change.

Summary
For seventy users, costs could break down as follows:

• 8 hours/day on-demand = $1950/instance over three years.
• 24 hours/day on-demand = $5295/instance over three years.
• Mixture of 50 reserved instances running 24 hours/day and 20 On-Demand Instances running 8 hours day = $3779/instance over three years.
• 24 hours/day reserved instances = $4511/instance over three years.

The difference in cost between on-demand usage 8 hours/day vs. 24 hours/day is enormous. Even the difference between 70 on-demand instances at an average of 8 hours/day compared to 70 reserved instances is huge: reserved instances are more than twice the cost. A mixture of reserved instances and on-demand usage probably won’t help enough to make it compelling. The only way that EC2 appears to be cost effective for large instances, used routinely, is to manage usage effectively. The detail for these calculations is provided below, with monthly costs. Those totals have been multiplied by 36 months and divided by 70 users for the cost summaries above.

An important note regarding Reserved Instances and cash flow
Reserved instances have an up-front cost of $910 per-instance per-year (or $1400 per-instance per-three-year-commitment) before usage charges are included (at a lower, reserved rate of $.24 per-instance per-hour). This means that reserved instances are a lot less viable for organisations looking to EC2 for cash flow benefits. The figures above were calculated using the $910 up-front cost, as I don’t believe most people will commit to three years of usage from the start. The reserved instance prices would clearly come down quite a bit with that three-year commitment, so feel free to recalculate as you like, but keep in mind that higher up-front cost is even worse for cash flow.

Cost assumptions

• 70 instances x 8 hours = 560 instance hours
• 560 instance hours x 230 days = 128,800 instance hours/year
• $0.48 per-hour per-instance
• Unattached Elastic IP Charges = $.01/hour unattached = 16 hours unattached/day/instance
• Elastic IP Address remap charges = first 100/month free, then $.10/remap
• EBS Storage = $.11/GB/Month x 50GB average storage
• Data Transfer In at $.10 per GB
• Data Transfer Out at $.15 per GB (first GB/month free)
• Reserved instance up-front costs of $910 for one year rather than the $1400 3 year commitment
• Bandwidth charges have not been calculated for VPC connections

Calculation Detail
On-Demand Costs at 8 hours/day average instance usage

• Instance cost = $61,824
• 16 hours unused Elastic IP charges/day = $.16 x 70 users x 230 days = $2576
• 1 remap/day x 70 users x 230 days = 16,100 remaps -1200 free = 14,900 x $.10 = $1490
• EBS costs:
  • Storage: $5.50/instance/month (assuming 50GB storage) = $840
  • IO: $5.50/instance/month (assuming roughly equivalent IO costs – perhaps 30-40 IOps) = $840
  • Snapshot Gets: costs should be negligible ~$100/year
  • Snapshot Puts: costs should be negligible ~$100/year
• Data transfer:
  • In: 20GB = $2.00/instance/month = $1680
  • Out: 20GB = $3.00/instance/month = $2520

Total = $71,970/year = $5997.50/month = £3793/month*
*This total does not include any support costs and is based on un-validated assumptions.

On-Demand Costs at 24 hours/day average instance usage

• Instance cost = $185,472
• 16 hours unused Elastic IP charges/day = $.16 x 70 users x 230 days = $2576
• 1 remap/day x 70 users x 230 days = 16,100 remaps -1200 free = 14,900 x $.10 = $1490
• EBS costs:
  • Storage: $5.50/instance/month (assuming 50GB storage) = $840
  • IO: $5.50/instance/month (assuming roughly equivalent IO costs – perhaps 30-40 IOps) = $840
  • Snapshot Gets: costs should be negligible ~$100/year
  • Snapshot Puts: costs should be negligible ~$100/year
• Data transfer:
  • In: 20GB = $2.00/instance/month = $1680
  • Out: 20GB = $3.00/instance/month = $2520

Total = $195,618/year = $16,301.50/month = £10,295/month*
*This total does not include any support costs and is based on un-validated assumptions.

Reserved Instances – Costs at 24 hours/day instance usage

• Instance cost ($910 x 70) = $63,700 + (70 x 24 hours = $92736 full-time usage) = $156,436
• 16 hours unused Elastic IP charges/day = $.16 x 70 users x 230 days = $2576
• 1 remap/day x 70 users x 230 days = 16,100 remaps -1200 free = 14,900 x $.10 = $1490
• EBS costs:
  • Storage: $5.50/instance/month (assuming 50GB storage) = $840
  • IO: $5.50/instance/month (assuming roughly equivalent IO costs – perhaps 30-40 IOps) = $840
  • Snapshot Gets: costs should be negligible ~$100/year
  • Snapshot Puts: costs should be negligible ~$100/year
• Data transfer:
  • In: 20GB = $2.00/instance/month = $1680
  • Out: 20GB = $3.00/instance/month = $2520

Total = $165,582/year = $13,882/month = £8771/month*
*This total does not include any support costs and is based on un-validated assumptions.

Mixture of On-demand and Reserved Instances – Costs at 24 hours/day instance usage
Mixture of 50 Reserved Instances and 20 On-Demand Instances.

• Reserved Instance cost ($910 x 50) = $45,500 + (50 x 24 hours = $66,240 full-time usage) = $111,740
• On-Demand Instance cost at 8 hours/day average instance usage = 20 instances x 8 hours x $.48 x 230 days = $17,664
• 16 hours unused Elastic IP charges/day = $.16 x 70 users x 230 days = $2576
• 1 remap/day x 70 users x 230 days = 16,100 remaps -1200 free = 14,900 x $.10 = $1490
• EBS costs:
  • Storage: $5.50/instance/month (assuming 50GB storage) = $840
  • IO: $5.50/instance/month (assuming roughly equivalent IO costs – perhaps 30-40 IOps) = $840
  • Snapshot Gets: costs should be negligible ~$100/year
  • Snapshot Puts: costs should be negligible ~$100/year
• Data transfer:
  • In: 20GB = $2.00/instance/month = $1680
  • Out: 20GB = $3.00/instance/month = $2520

Total = $139,550/year = $11,629.17/month = £7348/month*
*This total does not include any support costs and is based on un-validated assumptions.

Findings

Amazon Web Services can be quite expensive if usage is not controlled effectively. Based on these calculation, I don’t feel that $1950/instance over three years is bad value. These environments perform well and provisioning is very quick. There are no underlying virtualisation support costs or power costs. The scalability is inherently appealing and the Pay-As-You-Go model might be compelling for some businesses or independent users, despite all other considerations. In our case, we dove deeper in to the productivity question, attempting to get a handle on how the performance of server-class hardware in the cloud stacks up relative to laptops and desktop workstations. We wanted to understand if the complexity, potential remote worker issues and cost might be justified based on productivity gains. Some of these findings were surprising, as I will reveal in my next series of posts on SharePoint development environment performance.

Other posts in this series

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis
• Amazon VPC and VM Import Updates

Administration, Delegation and Usage Costs

The Tools

Unfortunately, the AWS Management Console user experience is fairly hideous. It doesn’t size properly in the browser and it has annoying synchronous post-back behaviours. It generally feels like an enormous Java app. I’m reminded of Cisco administration consoles circa the early part of this century. However, there is an Add-on for Firefox called ElasticFox which improves things a bit, but I wouldn’t say I’m thrilled with it either. I would classify it as less clunky, but I’d hesitate to go much further.

The AWS Management Console

ElasticFox

My colleague Brendan Newell co-evaluated Amazon Web Services with me. He identified we would need a more sophisticated management tool very early on. He found LabSlice and we looked at that for a bit. It’s fairly basic, but it adds some functionality that makes it compelling by comparison: policies, delegation and reporting. Those features provide administrative controls for smart delegation, or at least a start towards that control. It is a new product, so it’s reasonable to expect that it will improve. If we ever use AWS in anger, LabSlice or a tool like it will almost certainly form a part of the picture unless the Amazon administrative tools improve by then.

Why are these added features so important?

The underlying issue is that it’s more than twice as expensive to run an instance 24/7 than at 40 hours/week (at on-demand prices). Amazon provide Reserved Instances to try to address this always-on option, but the cost savings of “nearly 50%” assume the instance would always be running at On-Demand costs – so you’re paying for 50% of four times as many hours at On-Demand prices. This doesn’t really compute.

In reality, it may not be possible to run instances for only 40 hours/week, but it should be possible to run them for less than 50 hours/week for most users, with the right controls in place, and this figure could be a lot less if instances aren’t used every day.

So the question becomes how usage can be controlled without disrupting the value of the service. Too much control and the service becomes an obstacle to delivery. Too little control and the accounting department will be most displeased.

At a high level, these are the options we considered (with some bad options thrown in to illustrate the point):

• Get a reporting tool that will expose usage patterns on an individual and team level.
• Potentially bill teams for usage.
• Potentially bill clients for usage (trickier).
• Potentially set up a scheduled task that will automatically shut down an instance eight (or nine, or ten) hours after it is launched. Train users how to cancel the shutdown when they will be working late. While this solution is quite inelegant, it might work – depending on the users and their usage patterns.
• Use LabSlice (or a similar tool) to allow users to turn machines on and off, but not to create images or provision new machines. Set up policies to automatically shut down machines after a specific amount of running time.
• Get Draconian and have managers/administrators enforce shut down at the end of the day. Keep in mind, this is likely to taint any positives associated with this service and could prove very difficult to implement if users have valid reasons to leave machines on periodically. Is the enforcer really going to understand these nuances? In short, I suspect this won’t fit the culture of most businesses.

Remember, the point of all of this is to achieve the lowest cost, as the service will probably only be affordable with these controls in place. Without a mechanism to ensure machines are turned off, the business is exposed to 24-hour usage costs. I will give examples of projected costs without these controls later.

Back to the Public IP Addresses

Update 17 March 2011: the information regarding the public IP addresses and the VPC below is now out of date. Please see my follow-up post on Amazon VPC and VM Import Updates for more information.

If you recall from the last post, I mentioned that new public IP addresses are generated for instances whenever they are started up (unless the VPC is being used, in which case there is no public IP address). One of the features that you’ll want to find in your management tool is the ability to connect to instances after users have started them up. This environment isn’t going to work very well unless users can find out their new IP address every morning. As I mentioned before, this could also probably be scripted and is likely to form a part of other tools besides LabSlice. The point of reiterating this now is that it’s key functionality in a management tool and it will probably be very messy getting by without it.

Reporting

The last benefit of a good administration tool is reporting. If users are routinely forgetting to turn machines off, you want to know about it. If users aren’t using this system you probably want to know about it too. How are they circumventing this approach, and why?

I don’t think delegation can work without the reporting element, unless shut down policies are very effective and don’t cause disruption by terminating active sessions. Keep in mind that accountability is much less of a problem when clear, quantifiable costs can be attributed to actions. I think the ideal balance is probably high visibility of reports and delegation of start/stop functionality, potentially coupled with liberal shut down policies – perhaps at 12 hours of usage. Lastly, it should be clear that any of these approaches would need to be piloted.

Licensing

As mentioned in the first post in this series, Windows license costs are built in to instances and Amazon charges for instances based on the type of license they provide. The only license that must be paid for from Amazon is this Windows license and it is built in to the Pay-As-You-Go instance costs. If the instances are used for development then MSDN/Technet or other purchased licenses can be used in these environments for all licenses other than Windows, so long as the type of use is compliant.

Amazon offer an image with SQL built in to it. You will probably want to avoid use of this instance if you already have a SQL license, as it is considerably more expensive to run. The cost of a large Windows Server 2008 instance increases from $.48/hour to $1.08/hour accordingly. This is huge even if these numbers look small. There are 26,280 hours in three years. That’s more than $5,000 more expensive per-year (per-instance).

One thing that looked promising (until we realised it was only open to users in America) was the “Bring Your Own License” pilot. The program seems to be closed now, but I imagine this option would be interesting for readers of this blog, should that program ever form a core part of the offering, internationally. This of course assumes that subtracting Windows license costs from the instance charges results in a significant saving.

Recommendations

The main contentious issues for which there is no clear, one-size fits all guidance are topology, network configuration and management. We were looking at an all-in-one server, including the DC/DNS roles, on private and public dynamic IP addresses with considerable piloting in this configuration, supported by LabSlice. The costs of the management tool are going to be insignificant relative to what it saves you, even if you write it yourself. I consider it to be fairly indispensable, with the possible exception of Reserved Instances at the three-years up-front cost of $1400/instance (plus usage). I will explore these cost specifics in greater detail in my next post.

Other posts in this series

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis
• Amazon VPC and VM Import Updates

Machine names, Domain SIDs and Cloning

In our testing, we were able to run multiple instances of the same AMI concurrently, which can be desirable if you have a team of developers with similar or identical requirements. We could run these instances beside each other without conflicts because we had all roles (including the DC/DNS) on one machine and we locked down the firewall, which is advisable anyway in the cloud. We only allowed the RDP port inbound to start with, and opened HTTP/HTTPS traffic where it was helpful to do so. This cloning story would get much more complicated with multiple servers, as I discuss in more detail in the networking section below.

One big “gotcha” in this area is the default settings of the EC2 Service Properties when Amazon’s Windows AMI is launched initially. This is one of the few additions that Amazon packages with their Windows image. In the EC2 Service Properties you should de-select the Set Computer Name and Set Password options. The Set Computer Name option seriously causes problems for SharePoint, as it changes the Machine Name whenever the instance is started up. The good news is that you only need to do this once if you will be creating a new base image. Just be careful not to change this setting back later on.

The EC2Config Service

Networking

By default, Amazon assigns a public IPv4 address to EC2 instances via DHCP. This IP address changes whenever an instance is launched, allowing Amazon to manage their pool of public IPv4 addresses effectively. Until IPv6 adoption ramps up, this is the only viable option for an offering of this scale, although Amazon are actively looking at IPv6 today. By default, Amazon also assigns a private IPv4 address to EC2 instances via DHCP. This internal IP address also changes whenever an instance is launched.

Internal and external dynamic IP addressing introduces considerable design complexity for SharePoint development environments. This complexity is heightened by the addition of Elastic IP Addresses and the Virtual Private Cloud options.

Domain Controllers and Private DHCP

As I’ve mentioned before, SharePoint 2010 development environments need to be members of a domain in order to successfully deploy the Search or User Profile Service Applications, but unfortunately dynamically-assigned IP addresses and domain controllers don’t play nicely together. I shan’t delve in to those details much here, but this has been known to cause problems with start-up times for DCs, and member servers won’t know how to find the DCs once the DC’s IP address changes. Additionally, there are Firewall policy implications.

With the exception of the Virtual Private Cloud (discussed below), we had to rule out persistent multiple-server farms for these reasons. The complexities of managing this stuff on a daily basis would be beyond most users and would probably create system instability or at the very least, add cost (by leaving the DC on all the time). The option of adding a second DC for resilience and to possibly work around some of these issues would add further complexity and cost. Basically, this wasn’t working.

Developing on Domain Controllers

The only viable approach we could find for working with DCs on DHCP was to make the SharePoint development machine the domain controller. This is a step backwards in many ways, as this configuration has been known to cause issues. As I summarised in March 2010 (from the link above):

• Domain Controller security is bad for development. It means developers will be coding as Domain Admins and they will be doing so on a machine with Domain Controller security policies. This is just a mess. It’s tighter than it should be in some respects and looser in others.
• SQL doesn’t like to run on a DC.
• Running a DC, SQL and SharePoint on the same machine creates a massive load of service start-up contention and sometimes the environment will start from an unstable point because dependent services will not be ready when a depending service tries to start.
  • This also increases start-up time considerably.
• Adding Visual Studio to this mix causes known performance issues. The machine simply can’t keep up with doing all of this.

Having said all of that a while ago, based on lesser-performing equipment, we didn’t actually find that performance or installation were particularly troublesome on EC2, although we did encounter a security policy issue or two. I still have reservations about the code quality that will emerge from development on a domain controller, but if this is acceptable for your requirements then I think this is the most significant Private DHCP issue conquered. If not, you will probably need to look at the Virtual Private Cloud. Other topologies are conceivable but with even more complexity than we’re already contending with. These are unlikely to be broadly usable.

Public DHCP issues

The primary issue with dynamic public IP addresses is finding out what the new address is. This is easy enough if you have access to the AWS console, as you can pull the new address directly from the instance description and even download a file to launch an RDP connection to the new IP address directly. However, it’s very unlikely that it will be acceptable to give access to the AWS Management Console to all users. This leaves three options, as I see it:

• Leave the machines running 24/7 (at a potentially massive increase in cost).
• Have an administrator send/provide the addresses to users as the instances are started up.
  • This feels very clunky to me and untenable in the long term.
• Find a management tool (there are a few) or a scripted approach to handling this scenario.

Whatever the solution, it’s likely to form part of the broader question of administration, management tools and delegation, which I’ll come back to in the next post. I believe this can be solved without too much difficulty, but it requires some thought along these lines in order to avoid a mess.

Elastic IP Addresses

One way that Amazon has tried to ease the pain of Public DHCP is the Elastic IP Address. By default, each customer is given five, although you can request more. Elastic IP addresses are applied to an instance while it’s running. A few minutes after it has been applied it takes over from the DHCP-assigned address and users can access the instance at their usual address. However, this requires intervention by an administrator to associate the Elastic IP Address with the instance after it’s been started. Alternately it can be scripted. Just keep in mind this is another option that probably isn’t best delegated to everyone by giving all users access to the AWS console.

One thing that’s particularly crafty about Elastic IP Addresses is that you are charged $.01 for each hour they are not in use. If you’re diligent about turning off your machines when you’re not using them, you will get nailed for not using the IP address. Granted, it’s a small charge and with IPv4 address supplies dwindling very quickly, perhaps not that unreasonable.

In my view, Elastic IP Addresses probably aren’t going to solve a lot of problems, but in some cases it may make things easier – particularly if pointing DNS at these addresses.

Virtual Private Cloud

Update 17 March 2011: the information regarding the public IP addresses and the VPC below is now out of date. Please see my follow-up post on Amazon VPC and VM Import Updates for more information.

The Virtual Private Cloud (VPC) is effectively a VPN connection between your network and AWS. It allows fixed private IP addresses, DHCP options like DNS/WINS servers, and allows you to connect existing assets to the cloud, for instance management or backup servers. This may also help with SSO. I shan’t belabour the design options for the VPC, because at face value it should be pretty obvious if it’s the right fit for your uses. There are obvious security considerations about opening up this communication across the WAN to a third-party, but that’s not to say there aren’t ways it can be set up well – for instance creating a dedicated domain in the VPC.

The most important thing to know about the VPC is that when instances are launched they only get a NIC with an IP address on a VPC subnet. There is no public IP address for the instance. This means the only way you can access the instance is via the other end of the VPN (typically the corporate network). This may introduce some funky routing and potentially degrade speed/reliability for users working from home or on client sites. On the other hand, it may not. It’s critical that this option is thought through with a broad design team including internal network and systems teams. I would highly recommend testing/piloting this configuration as well (noting that the initial configuration may be expensive for a test, since it will integrate with production infrastructure).

I think the VPC can answer a lot of the shortcomings of the standard EC2 IP addressing approach if public IP addressing is not a requirement. I’m not sure why NAT couldn’t have been used to allocate fixed internal addresses by default, but it hasn’t been, so we’ve only got one way in to the VPC. Once in it, you can deploy single-server machines as we did without the VPC (assuming the firewall is locked down in the same way), or join SysPrep’d SharePoint servers to a shared domain infrastructure. This assumes SharePoint provisioning (scripting installation/configuration) is mature enough that manual configuration steps don’t impede productivity. Other topologies may be valid as well. In principal it shouldn’t be miles different from your LAN. The main things to understand is that the private IP addresses are assigned by Amazon and there is just the one way in. Note: there’s quite a bit to understand about planning the VPC itself, and pricing for that traffic, which is all outside of the scope of what I’m inspecting here, so please refer to the Amazon VPC resources for more information. Also be aware that it’s still in Beta.

Networking can be enough to melt anyone’s brain, so I’ll save administration, delegation and licensing until my next post.

Other posts in this series:

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis
• Amazon VPC and VM Import Updates

What are the Amazon Web Services?

AWS is a platform in the cloud, like Windows Azure in some respects. While these web services are distinct from traditional hosting offerings, Amazon also provides Infrastructure as a Service (IaaS) in the form of Elastic Cloud Compute (EC2). This is a Red Hat implementation of the Xen hypervisor, from which virtual machines (instances) can be launched. For accuracy, I should note that Amazon recently launched a second Oracle hypervisor within EC2, but that’s a distraction from this discussion. Amazon have been providing their web services since 2006. For the purposes of these posts I am concerned with the EC2 offering as a cloud-based alternative to desktop development workstations, although there are other scenarios that may be suitable for deployment in EC2, such as demonstrations or large infrastructure tests. For more information on the difference between traditional hosting and EC2, see Amazon’s FAQ on the matter.

What is Elasticity?

This term arises frequently in the Amazon vernacular. In its essence this means that scalability is built in to the platform. Need more CPU or memory? Just re-launch your instance as a larger size. Need more instances? Create them in a few minutes. Need more storage? They got it and then some. IP addresses? They even have Elastic IP addresses. Bandwidth? It’s the cloud, fool.

AWS largely deliver on these promises, although you’ll encounter some provisioning fiddlery before realising it. More importantly, increased size comes at a cost. Nearly all of the Amazon price points are ridiculously low at their smallest, but these costs are not always linear – particularly with CPU and memory. Additionally, cost permeates nearly every design option, and these costs persist over time. Infinitesimally small prices need to be considered over very long periods if IaaS is to become an alternative to hardware. I will discuss costs in more detail later, as this topic is fundamental to the desirability of cloud computing. If it isn’t cost effective, it probably won’t be the right option. But in a nut shell, Elasticity means that if you need more of anything, you can pay for it for the duration that you need it. They definitely intend to say that you can shrink as well.

EC2 Design Complexity

I couldn’t possibly hope to explain everything that’s important to know about AWS in these blog posts and I won’t try. However, it’s important to know that the design constraints that pricing and scalability impose on AWS require a fresh perspective.  Infrastructure Architecture for AWS will require time, testing, piloting and a good understanding of end-user working patterns. Once this configuration and these patterns are clearly understood, the costs need to be projected over long periods. This is likely to be a deep consulting exercise, since so few design options can be left to chance; this will hopefully become clearer as I talk more about pricing later. For now, if you don’t believe this is complicated, have a look at the 237 page User Guide, which I would class as required reading for anyone serious about EC2. The topics covered below are a summary of the areas that I feel are most important to understand with SharePoint on EC2.

Storage

The first thing to understand about AWS is that there are two types of storage, the Simple Storage Service (S3) and the Elastic Block Store (EBS). Some older documents and forum posts were written before EBS was available as a root device, so watch out for potentially misleading information.

All SharePoint 2010 environments need to run on EBS because Windows Server 2008 will chew up more than 10 GiB off the bat (this is the maximum size of S3 volumes). EBS storage costs are more expensive than S3 and you pay for the number of I/O requests, so projecting costs is a fairly inexact science. However, in my brief testing time the I/O charges were relatively small. It’s worth noting that for the extra cost, EBS volumes also persist and they launch faster. I am only briefly touching on this topic, so please review the User Guide if this is insufficient detail. The key points for now are that you must use EBS for SharePoint instances, and EBS is more expensive than S3.

Provisioning
Taking snapshots and creating new images from them is quick and easy in EC2, once you get your head around the key concepts: AMIs, Volumes and Snapshots.

AMIs
An AMI is an Amazon Machine Image. This will be the first design choice you encounter when launching an Instance. Amazon provides a basic Windows Image or you can use an Amazon image with SQL included (at a cost). You can use your own licenses for everything but Windows.

Once an image is running you can modify it to your taste. Once you’ve created a new standard baseline, you can create a new image from your instance, and when you provision new instances you will be able to select this new image rather than the Amazon one you started with. Note: the Amazon Windows license cost is built in to the billing process; your instance costs include the license, even after you’ve created your own new image from the original. Also note: Windows Server 2008 R2 is not available yet.

Volumes
A volume is basically a virtual hard disk. When a new instance is created, the selected AMI is deployed to a new volume – the same size as the image it was created from. A volume can only be attached to one instance at a time, but an instance can have many volumes attached to it if you want to add storage capacity.

Remember that you pay for the storage you use, so size your volumes wisely. 30 GiB is unlikely to last anyone very long with Windows Server 2008, so consider at least 40, if not 50 GiB for any new root volumes. Keep in mind, you may find the less expensive S3 volumes useful as secondary, disposable storage if that suits temporary needs.

Snapshots
A snapshot records the state of a volume at a point in time. Once a snapshot has been taken, a new volume (of equal or greater size) can be created from the snapshot and that new volume can be attached to a new instance. That new instance can be used to create a new AMI at the new size. Snapshots and new volumes together enable you to increase system disk size. Snapshots can also be used for backup.

An example workflow for getting your first image at the right size might go like this:

• Launch an instance from the default Windows Server 2008 image.
• Install SQL and SharePoint (this should be possible at just under 30GiB).
• Configure stuff and shut down the instance.
• Take a snapshot.
• Create a new volume at 50GiB based on the snapshot.
• Detach the existing volume from the instance and attach the new volume.
• Create an image from the instance.
• Launch the existing instance and create additional instances from the new AMI as needed.

Note: if you will be including Visual Studio or any other sizeable software, you will need to go through a process like this before installing it, as it will push you over the 30GiB initial size.

This process is oversimplified, but it hopefully illustrates the relationship between AMIs, snapshots and volumes as they relate to provisioning. All told, I think this way of working with images, volumes and snapshots is sensible, not terribly complicated in the EC2 scheme of things, and the choices should be pretty straight-forward once you understand the options and costs. However, this could potentially get more complicated as end-users engage with these decisions. How will they know what to ask for? Will it be necessary to involve EC2 experts in the approval of any new systems? Training, consultancy or winging it all have associated costs and risks. Even though I’m only talking about development environments here, there are still risks  in committing to a Pay-As-You-Go platform where usage is unrestricted. Keep this in mind.

I’m aware this is lengthy already, so I’m going to split this up. In my next post I’ll review Cloning and  Networking.

I still have yet to find the time to test this myself, but I’d be very keen to hear about your experiences – good or bad. Failing that, I hope to get back to this in the next couple of weeks.

“…an IIS 7.0 extension that meters the download speeds of media file types and data between a server and a client computer. The encoded bit rates of media file types such as Windows Media Video (WMV), MPEG-4 (MP4), and Adobe Flash Video, are automatically detected, and the rate at which those files are delivered to the client over HTTP are controlled according to the Bit Rate Throttling configuration.”

It basically saves you bandwidth by only transferring what you’ve watched plus a small, configurable buffer. Think about each user that starts watching a ten minute video but only watches one minute. In that time, they may have downloaded five minutes of content – quadrupling the bandwidth consumption unnecessarily. Bit Rate Throttling shares some user experience characteristics with Streaming Media, but it works on a normal web server over HTTP. It’s really quite a simple tool and I won’t devote space here to explaining it when the IIS.NET site already has some great content, including a brief introductory video. Definitely check it out.

So why am I writing about it?

• It’s cool! We wanted to turn it on to better manage bandwidth while delivering video with SharePoint 2010′s Silverlight Media Web Part.
• Unfortunately it didn’t work when we turned it on. In fact, it crashed our w3wp.exe for the web application where it was enabled.

At the time, I reported the issue on the SharePoint 2010 TechNet forums and IIS.NET, but didn’t get very far. We eventually decided to live with it, leaving BLOB Caching on, even if the bandwidth was left unoptimised.  To summarise the fault, when Bit Rate Throttling was enabled, my web application would load a page or two, then that application’s w3wp.exe would (apparently) leak memory until it crashed. Repeat. The issue and my troubleshooting is explained in more detail on those two threads. These are the key error messages:

Faulting application name: w3wp.exe, version: 7.5.7600.16385, time stamp: 0x4a5bd0eb
Faulting module name: bitratemodule.dll, version: 7.1.625.10, time stamp: 0x4aca8535
Exception code: 0xc0000005
Fault offset: 0×0000000000007669
Faulting process id: 0×2140
Faulting application start time: 0x01cb2cd16410dafa
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Program Files\IIS\Media\bitratemodule.dll
Report Id: e3004397-98c4-11df-91ae-00155d06ab22

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: w3wp.exe
P2: 7.5.7600.16385
P3: 4a5bd0eb
P4: bitratemodule.dll
P5: 7.1.625.10
P6: 4aca8535
P7: c0000005
P8: 0000000000007669
P9:
P10:

Attached files:

These files may be available here:

Analysis symbol:
Rechecking for solution: 0
Report Id: e3004397-98c4-11df-91ae-00155d06ab22
Report Status: 0

Luckily, I noticed that Jack over at IIS.NET recently posted:

I believe we have identified the problem and have a fix for the issue.  The fix will be available as part of IIS Media Services 4.0 which will be released in the very near future.

Good news! I see that a release is now available for IIS Media Services 4.0 Beta (second down in the right-hand column on the Bit Rate Throttling site). Unfortunately I haven’t had a chance to test this yet and I’m not sure when I’ll get the time. For now, I’m posting this incomplete, as it would be great if many people tested this and the IIS team got as much feedback on this technology as possible while it’s in a Beta release. Obviously, I’d caution against installing either version in production for now. 3.0 doesn’t work and 4.0 is very new.

A few related notes:

• BLOB Caching is not a requirement for Bit Rate Throttling in general, but it is a requirement for Bit Rate Throttling SharePoint 2010 web applications. The Plan for caching and performance document notes, “Bit rate throttling will not work correctly if you do not first enable the BLOB cache and configure it to cache the files types that you want to throttle.”
• We didn’t look at Smooth Streaming for very long, because the Silverlight Media Web Part hasn’t been built to adapt content in that manner.

The machine-default permission settings do not grant Local Activation permission for the COM Server application with CLSID
{000C101C-0000-0000-C000-000000000046}
and APPID
{000C101C-0000-0000-C000-000000000046}
to the user <FARM ACCOUNT> SID (S-1-5-21-xxxxxxx….) from address LocalHost (Using LRPC). This security permission can be modified using the Component Services administrative tool.

The CLSID for this COM Server Application is MSIServer, used to activate the Windows Installer Service. You can find this by navigating to HKCR\AppId and examining the details there:

Given that there were 105 instances of this DCOM 10016 error in an eleven second period, I decided to see what was happening at the same time (00:52:09-00:52:19) in the Application event logs. It turned out that there were 210 Information and Warning events during the same time-frame. An example pair of these event is included here:

1035 Information
Windows Installer reconfigured the product. Product Name: Microsoft Excel Mobile Viewer Components. Product Version: 14.0.4763.1000. Product Language: 0. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 0.

1015 Warning
Failed to connect to server. Error: 0×80070005

You might notice this first informational event is for the Microsoft Excel Mobile Viewer. As you trawl through the events it will become clear that these events occur for Language Packs, Service Applications, Foundation elements, Web Apps – all sorts. At this point it was pretty clear to me that the SharePoint Farm account (probably a Timer Job) was trying to run the Windows Installer Service for these products, but I had no idea why, so I cracked open ULS Viewer and went to town.

A Timer Job called job-admin-product-version kicked off at 00:52:08.92. Filtering by that CorrelationID I could see that this job appeared to build a hierarchy of upgradable elements (Foundation stuff, Service Applications, etc), then checked to see if these elements can be upgraded. But I don’t really feel comfortable trying to figure out what a Timer Job does by stepping through ULS logs, so before going any further I had a look at the TechNet Timer Job Reference and found that Product Version Job runs nightly at 00:45 by default.

Now that we know that this job “checks the install state of the machine and puts that data into the database“, I’m going to take a leap of faith and assume that the farm account is trying to use the Windows Installer Service to do this. Hopping back in to the ULS logs, the next-to-last event correlates to the deluge of Application logs entries.

Note that one of the first items in this mess of updates is Microsoft Excel Mobile Viewer Components again.

UpdateProductInfoInDatabase, regProductsQuery = exec proc_RegisterProductVersion N’20c667df-1bc3-486b-869c-a3ba40f83af5′, N’Microsoft SharePoint Server 2010′, N’14.0.4763.1000′, N’{90140000-1138-0000-1000-0000000FF1CE}’, N’Microsoft Excel Mobile Viewer Components’, N”…

It seems pretty clear that Product Version Job checks the installed versions of SharePoint Products and Technologies, then updates that info in the database (presumably Central Admin config). However, it’s not really clear when that information gets used, so changes to the default Daily job schedule may have unintended consequences. As I see it, this leaves three options:

• Live with the warnings/errors until a better option becomes available.
• Disable the Product Version Job timer job, noting that this could potentially have a negative impact on updates to the system (not recommended until these implications are better understood).
  • Potentially combine this strategy with a plan to make the farm account a local admin temporarily and run the job manually at routine intervals (again, this warrants testing and a better understanding of the Timer Job itself).
• Grant the WSS_ADMIN_WPG local group or Farm account permissions to Launch and Activate the Windows Installer Service (which I don’t recommend).

I’m presently contending with these errors in a development environment, so I’m going to live with them for now. I’m pretty reluctant to recommend the last option. It seems to me that if the Farm account has rights to elevate to Local System via the Windows Installer Service, that puts a pretty big dent in the least-privileged model. I’ll keep looking in to this, but I thought I’d identify my findings so far and I would welcome any comments or ideas that I haven’t considered – particularly if anyone has more information about when this database information is used.

Update: 20/2/2011

I’ve done some more work on this now.

Inside Manage Patch Status

Testing Manage Patch Status

I hit the search engines in earnest and found that these problems were prevalent across a fairly wide range of graphics cards. We enlisted Dell’s help and they told us that they do not certify that Hyper-V will work on any laptops. More precisely, they clarified the primary support concern is that future driver releases may not work with Hyper-V even if we find a model that works with today’s drivers. At this point we were considering a pricier Precision model and they put us in touch with their Precision product team in Texas. They were most helpful but we were told that Dell themselves do not use Hyper-V on laptops except for demonstration purposes and they simply use it as a server for connected workstations, so they would never experience the same graphics issues. Dell kindly offered to let us test our development build on various models at their campus if we agreed to share the results with them, but before we could arrange that visit, Windows Server 2008 R2 SP1 Beta* was released and I upgraded my machine in order to test out Dynamic Memory.

As I was installing it I had a chat with my colleague (and serial early adopter) Lambros Vasiliou to gauge his impressions. He mentioned his favourite improvement is that the known Hyper-V host graphics performance issues are either gone or greatly mitigated. This is an issue that’s been repeatedly discussed in our organisation since we moved from a hotchpotch of virtualisation technologies to Hyper-V as our standard development build last year. It’s probably the single thing that irritates our users of this system more than anything else.

I did some testing myself with videos playing and moving windows about with Windows Key + Arrow hot keys. The results were fairly impressive – without doubt a big improvement. One thing that still behaved poorly on my Dell XPS M1330 (with NVIDIA GeForce 8400GS) is full-screen YouTube, Vimeo, etc. The CTRL+ALT+DEL redraw operation seems a bit sluggish still as well. I noticed that my PowerPoint Presenter View was better, but still not 100% responsive.

I also tested on the Dell Latitude E6410 (with NVIDIA NVS 3100M). Not only is the previously-mentioned blue screen fixed and the graphics generally improved in the same ways as on the XPS, but the full-screen in-browser video and CTRL+ALT+DEL are instantaneous. One possible explanation for this different experience is that the Latitude has a processor with SLAT, but I can’t validate that at all yet… because I can’t find any information whatsoever about why/how this has changed!

I think it’s unlikely that these changes are related to RemoteFX (since the XPS M1330 does not have a processor with SLAT and I never enabled it on the Latitude E6410). I would expect RemoteFX to improve the experience connecting to the guests, not the Hyper-V root partition (although it’s possible that this improvement is somehow related). I’ve tried pinging Virtual PC Guy and posted this query on the SP1 Beta TechNet forum but so far the community can only confirm that this is indeed working on a number of different models including a Mac (drill down in the links on the TechNet thread for more information). One way or the other this is great news, but I’m finding the lack of information about these changes quite maddening given the amazing detail that’s been produced for the Dynamic Memory launch. I’d really appreciate further insights if anyone can reveal the internals.

* A few notes regarding the Service Pack 1 Beta installation process:

1. The links on the SP1 Beta page are a bit confusing. You should be aware that if you click the “Evaluate Windows Server 2008 R2 and SP1 Beta” link you will be taken to a page with a “Download Windows Server 2008 R2 Trial Software” section at the top. “Download SP1 Beta Software” is beneath that section. This is what you want. If you click the first link you will initiate a download of the full Windows Server 2008 R2 (SP0) installer. If you “upgrade” your system using that installer you’ll wind up with a nice new trial version of SP0. AGH. Starting again from the links in the right section I was able to run a small installer that presents the updates to Windows Update and that has all worked fine, so I’d recommend that route. Alternately the Service Pack can be downloaded stand-alone. I did that for my second install and it worked fine too. Also note the Windows Server 2008 R2 SP1 Beta Reviewer’s Guide, “to evaluate the core features of Windows Server 2008 R2 SP1 Beta release in your environment”.
2. If you use Forefront you will need to uninstall it in order to install SP1 Beta, so make sure to remember to reinstall it afterwards.
3. When I installed the Service Pack my screen went black for about ten minutes following the first reboot. Be prepared for this. You’ll see plenty of ongoing disk activity but nothing on the screen. Fairly disconcerting, but perhaps this is all a part of these same video changes.

• Microsoft’s approach to Dynamic Memory is fundamentally different than VMWare’s overcommitment, in that VMWare doesn’t trust information about memory usage from within the guest, whereas Microsoft’s implementation is based around an awareness of the amount and type of memory that’s being used at all times.
• Dynamic Memory will work by Adding/Removing memory.
  • Adding memory is enabled through a new synthetic memory driver.
  • Removing memory that’s not being used is done with ballooning.
  • Memory is now assigned with a few new values:
    • Startup memory is the amount of memory assigned to a VM, which is also the minimum memory the VM will consume (default value is 512 MB).
    • Maximum memory limits how much memory a VM can consume.
    • Priority can be assigned to specific VMs in order to make sure that they receive available memory before other lower-priority VMs.
    • A Memory Buffer can be set to reserve memory for specific VMs, for instance if they need extra memory for file caching.
• Hyper-V Manager adds two new columns.
  • Current Memory identifies how much memory the VM is consuming.
  • Memory Availability identifies the difference between how much memory a VM has vs. wants in a +/-% figure.
    • When the availability goes negative, the Windows guest will start to work with the lesser amount of memory that’s now available to it (via paging, etc).
    • Negative availability will result in reduced performance, but the systems will continue to function.
• Memory is now reserved for the root partition in a different way, so that dynamic memory won’t bring down the host.
  • This amount can be configured with a new registry key based on how the root partition is being used, for instance if it’s your desktop OS.
• As Dynamic Memory is used more, the chances of spanning NUMA nodes increases (on NUMA systems).
  • He points out that different systems have vastly different Back Channel performance, so the impact of NUMA Spanning can be negligible or drastic.
  • In SP1, NUMA Spanning can be disabled (if desired).
• Dynamic Memory also supports Large Pages, which are likely to become more common with virtualised Exchange/SQL.
  • VMWare cannot overcommit these pages.
• I’ve asked if there are specific processor requirements. I’ll be interested to see how/if this supports processors that don’t have SLAT.

When I initially created our SharePoint 2010 beta development environments, I built them in a Workgroup, for many of the reasons that I chose to do so in 2007 (see the Workgroup Development section in Part II of my series on SharePoint development environments). Now, the SharePoint 2010 Configuration Wizard (psconfigui) throws an error when choosing the Complete install option in a Workgroup. I was able to get past this error by following the suggestions on the Microsoft From the Field blog, but a few weeks down the road we’ve pinned down two issues (as confirmed by Neil, the author of that post).

1. Search will crawl successfully in this configuration, but the query role will never initiate. Queries from web applications that consume this Service Application will produce an error, which you will be able to trace to an application event log 6398 error and the key event in ULS, “The SDDL string contains an invalid sid or a sid that cannot be translated.” As Victor Magidson from Microsoft puts it on this Technet thread, this error occurs when a, “topology activation job is trying to create a propagation file share.” In short, the query role never finishes initialising so it will never serve queries. This only occurs on the single server Complete installation and deleting/re-creating the Service Application yields the same result.
2. The User Profile service application will also be useless*, but it always would have been since you couldn’t import users from the local user database in 2007 either. However, this has wider implications in SharePoint 2010 because of the social computing features, which developers probably won’t be able to live without.

All this means that I need to consider rebuilding our development environments in a domain or we need to use the Simple installation. In the past I always avoided the Simple installation because it uses system accounts like Network Service for application pool identities, so it took some re-acquaintance to identify some of its limitations. It has no configuration options (thus the use of local system accounts), the User Profile Service Application does not start (presumably per Forefront Identity Manager 2010 installation issues as seen in other topologies) and it uses SQL Server Express (which developers aren’t very fond of), so we’re unlikely to pursue this option. Accordingly, we’re reviewing the ways that we can re-create the development environment in a domain.

Why don’t I just make the development machine a domain controller? This isn’t clear-cut and there are benefits to the simplicity of it, but ultimately I think it’s a bad idea because:

• Domain Controller security is bad for development. It means developers will be coding as Domain Admins and they will be doing so on a machine with Domain Controller security policies. This is just a mess.
• SQL doesn’t like to run on a DC.
• Running a DC, SQL and SharePoint on the same machine creates a massive load of service start-up contention and sometimes the environment will start from an unstable point because dependent services will not be ready when a depending service tries to start.
  • This also increases start-up time considerably, which is a big concern when using Hyper-V on a laptop, where we don’t have Sleep/Hibernate.
• Adding Visual Studio to this mix causes known performance issues. The machine simply can’t keep up with doing all of this.

Why won’t we use a central development domain? Because then we can’t clone the VMs. We don’t just clone standard builds, we also use the same VM for each developer/consultant/architect on a single project. We may opt for this approach eventually, but I think the investment in automation is pretty considerable for it to be efficient with our team development requirements.

So I’m probably going to separate the DC on to a different server for now. I’m unlikely to add a third server for SQL, as even though it might be faster, I think the complexity of managing networking and snapshots across three servers is undesirable for developers, if we can get by with two.

All of this underlines the reasons why I didn’t publish any SharePoint 2010 build guidance yet. Some things don’t become clear until you get your hands dirty. I’ll revisit this topic as soon as I feel comfortable that we’re pinning things down, but for now we’re still adapting our methods.

*I’ll note that I haven’t properly tested Claims Based Authentication against a single server complete installation, so it’s conceivable that you could import non-A/D users in to this configuration, but you’d still need to live without Search.