SSD

Probably the most contentious finding from my initial testing was that disk performance and bus speed aren’t significant factors in most of those results (start-up and shut-down times being a notable exception). To recap a bit of my initial summary:

Disk performance and bus speed did not prove to be significant factors in these results (except for virtual machine start-up times). Obviously there are fundamental differences about SSD (yet untested) that may skew this picture, but I will be surprised to see big differences. If we’ve got these tests right, and they are actually representative of the tasks that slow down development, then we would expect to see wider variance across bus or disk speeds. We don’t.

• This assumes the disk is relatively uncontended. Virtual machine performance degrades in every type of test while large file operations are running concurrently on the same disk. This could be copying an ISO, importing or exporting a virtual machine or any other sustained large file operations.

The obvious follow-on test would be to repeat on the same system with SSD. Unfortunately I’ve not found the time or hardware resources to do that yet, but today I ran an indicative test. In this scenario, I installed two new boots on a brand new Lenovo ThinkPad W520. One drive was an SSD in the second bay, the other was a 7200 RPM SATA drive (I don’t have specs for either to hand, but they were the default Lenovo offerings). For both boots I ran the VMs on the other spindle, so we had one test with an SSD system drive and VMs running on mechanical drive. For the second test I inverted the configuration and had a mechanical system drive with VMs running on SSD. In both cases there was no appreciable system contention outside of these tests.

The results? Identical. First page load after application pool recycle times were around 10 seconds for Central Administration, a blank site and a My Site Host. 16 seconds for a customised intranet solution (the same one from the initial tests). These are very similar times to the desktop results from my original tests – only marginally slower. What does this tell me? I should complete the testing for more scenarios than just the first page load times. But given that it won’t happen any time soon, I’m pretty comfortable assuming that SSD isn’t going to automagic performance improvements where disk speed is otherwise not a factor, and I’m happy standing by my initial analysis with this supplementary finding in hand.

To be crystal clear, I’ve seen first-hand how quickly a VM starts and shuts down when running on SSD. It’s stunning. And there’s clearly a subsequent gain reaching a post-start-up stasis of sorts. I always waited for my system to calm down like this before any testing could begin, and in some cases that might take ten minutes on a 7200 RPM mechanical drive, and even longer over USB2. However, I don’t actually see this as a major productivity loss. Irritating, yes. A sound business case? Probably less so. I imagine doing lots of full crawls would translate to a big productivity gain on SSD, but is that a major issue for most developers on most projects? Not consistently so, in my experience. But if you’re developing a FAST solution it would probably be a good idea. Maybe even isolate all of the DBs on the SSD. There would certainly be scope to play with this once you have known disk contention that you’re fighting.

The problem I have is that I can’t find any other scenarios which are as disk-bound as we might assume. When we first started this testing in late 2009, our first inclination was to add eSATA drives on a PCI Express port to get a second spindle. Freeing up the VMs from system activity and large file operations on the system disk is a clear win, but this will be true for any disk of any speed on virtually any bus if my initial test results are to be trusted, which means that the SSD investment for VM performance gains is only likely to get you faster start-up/shutdown times and anything else that involves large file operations.

All this said, if budget and SSD reliability are not concerns, load up on them, assuming it gets you sufficient storage capacity. It won’t hurt, so long as they don’t fail all the time. Additionally, it may be beneficial to get an SSD for the system drive, if other non-development activities would benefit from it. Or it may be that start-up/shutdown times are compelling on their own. In the final analysis, I’m in no way opposed to SSD, but when it’s my neck on the line for justifying hardware purchases, I want concrete, consistently-realised performance gains if I’m going to recommend a less resilient, lower capacity, more expensive technology. In most cases, I’m not sure that’s the case for virtualised SharePoint development.

i5 vs. i7

One of the other key follow-on investigations from last year’s testing was a comparison of i5 vs. i7 processors. I’ll quote the initial context here:

• The benefit of spending on i7 processors is in doubt. We are seeing very minor performance penalties when adding more than two CPUs in both VMWare Workstation and Hyper-V for most tests. There were also very minor improvements for some tasks, but on the whole there does not appear to be a measurable benefit. This might vary if the host OS is doing a great deal with the CPU, but that is liable to cause other contention issues than just in the CPU (on a laptop).
• The only tasks that appeared to use all 8 cores in a SharePoint VM were:
  • Retract/Deploy of a solution (but only very briefly)
  • Create web app, or Create site collection (but at low percentages)
  • Rebuild with Code Analysis (but not fully)

Since the initial testing, I’ve continued to experiment with two versus four cores in the VM, and have never seen a significant enough difference to endorse using more than two, but at the same time, I don’t think the penalties for multiple cores are significant enough to worry about, if any user thinks that four cores will be better. Note: I’m only talking about development here.

Based on these findings, I had a hunch that a faster clock speed i5 would outperform an i7, assuming two or fours cores running inside the SharePoint VM. For the sake of simplicity I’ve tested with two cores. For these follow-on tests I used the same ASUS V6-P7H55E model that I used during the original testing, with an identical spec/configuration and the same VM, with one exception. We replaced the Intel® Core™ i7-870 Processor (8M Cache, 2.93 GHz) with an Intel® Core™ i5-680 Processor (4M Cache, 3.60 GHz) – faster speed, smaller cache.

To my surprise, the performance tests returned virtually identical results to my initial testing (all within the margins that the initial tests deviated). Reviewing those results again, we can see that for most tests disk performance is not an issue (see above), and these tests suggest that CPU is not a bottleneck to further performance gains beyond a certain point (I believe an older CPU would fare poorly against either of these, but if a 4.0 GHz i5 came along, I’m not sure we’d see an improvement over these results). These machines have reasonably high-spec RAM, so memory speed does not seem a likely candidate for further improvements. Based on resource monitoring during testing, I can’t see that anything is maxed out, so I’m beginning to think there’s something inherently languid in the sequence of this computation. Perhaps a deeper dive is in order some day, but I’m probably not the best person to take that on.

As an aside, I can confirm that I’ve been running up to six VMs concurrently on this desktop with the i5 over the last couple of weeks. Starting all of the machines up at once is rough, but after 15-20 minutes it’s handling it no problem, and I don’t have to do that unless I’m taking a major snapshot. This suggests disk starts to become an issue with six VMs running at once, but that shouldn’t surprise anyone. If anything I’m surprised it’s not more of an issue on this machine, and if I continue to need this many VMs at once I’ll probably sacrifice my RAID 1 array for two separate disks. I’d be hesitant to suggest SSD in this case, since six VMs is probably going to chew up more storage than most SSDs will accommodate.

Based on these results and this longer-term experience, I’d recommend the higher-speed i5. I don’t seem to lose anything with the i5, at any rate. Maybe even go down to a 3.0 GHz i5 and save some money? If you know you have a specific scenario that will consistently utilise eight cores, go for the i7. But ultimately, both of these CPUs are fast.

Windows Experience Index and CPU Benchmarks

I’ve had a number of discussions with people about performance since publishing these posts, and it’s surprised me to find how many people actually look at the Windows Experience Index. Unfortunately, in my experience, this really doesn’t tell us much on today’s machines. A poor-to-average developer machine today gets a good score, unless it has a 4200 RPM hard drive (in which case it shouldn’t be used by a developer). Also, graphics performance is probably irrelevant. I really don’t think this index sheds any light on the SharePoint Development Experience Index, as it were.

Along these lines, with the receipt of a few new Sandy Bridge CPUs in these Lenovo laptops, we started running CPU benchmark tools. These are quite useful for diagnosing problems (early BIOS versions on the W520 were slooooooooooooooooooow – make sure to apply v1.25+), but beyond that, I’m not sure they tell us what we need to know for SharePoint development. For instance, at one point we saw hugely different CPU benchmark scores but the SharePoint performance tests were roughly the same. I guess I mention these tools here to say that they may be useful in some cases, but I think these real world tests probably tell us more.

Sandy Bridge

“How good are these Sandy Bridge CPUs”, I hear you mutter? Battery life is amazing. I accidentally left the Lenovo W520 running unplugged all day today. I think it lasted about six hours. Performance-wise, you’ve seen the ~10 second first page load times after an application pool recycle on a few of the standard SharePoint OOTB templates. That’s on a 2.0 GHz Sandy Bridge i7. This is not far off the 2.93 GHz first generation i7 desktop CPU results from our original tests, and much better than the first-generation i7 laptop PCU. Pretty good, I’d say. I can’t wait to see the second generation desktop speeds. Note: the desktop i7 models also have integrated graphics, where the first-generation desktop i7 CPUs did not. Now to hope there aren’t any more recall issues.

A few other things to note:

• Until a few months ago, I didn’t realise that dual-core i7 CPUs exist. I thought they were all quad-core. Not so. This is important because if you find a laptop model with four SODIMM slots (to get you 16GB RAM, and 32GB in due course), the fine print will probably tell you that you will only get two SODIMM slots unless you purchase a quad-core CPU.
  • There’s a secondary “gotcha” here, in that the quad-core laptop i7 CPUs peak at a much lower clock rate than their desktop siblings. I think the fastest first-generation quad-core laptop i7 CPU peaks at just over 2 GHz, and most laptop manufacturers have very few models, if any, with this CPU. In fact, we struggled to find anything other than 1 Lenovo, 1 Dell and 1 HP model at 15″. Most of these only had availability for lower clock speeds and we nearly had to settle for 1.73 GHz. These are all very expensive as well.
• The Sandy Bridge comes to the rescue here insofar as it has higher clock speed quad-core laptop i7 models, even if these are also slower than their desktop siblings. However, it’s also worth noting that in most CPU comparisons of Sandy Bridge to 1st-generation i7 models, the Sandy Bridge annihilates. Basically, at this point, if shopping for a high-performance SharePoint development laptop, you should be looking at Sandy Bridge. They may actually be cheaper as well – somehow.
• Also be aware that the Japanese earthquake has caused severe manufacturing delays for most hardware vendors. You may find you need to settle for a lengthy lead time at the moment.

The page cannot be displayed because your server’s current configuration does not support it. To perform this task, use the command line operations in Stsadm.exe.

How odd, given the emphasis on PowerShell in SharePoint 2010! After a bit of head scratching and examining application and ULS logs, I navigated to the Central Admin home page and everything appeared to be fine, but then when I got around to creating a new Site Collection a bit later, I got the same error, even though I was able to create web/service applications. I had the same error when logged on as farm admin, farm admin + local admin rights, farm admin + SQL SysAdmin and farm admin + domain admin rights, so I was pretty sure it wasn’t a permission issue (and I should note my temporary fiddlery here is only really suitable for non-production environments). This error also occurred on some other Site Collection-specific pages.

After searching for a solution I found a number of suggestions that this was related to insufficient rights for Active Directory Account Creation Mode. So I played around with SQL permissions/accounts a bit more and was eventually able to loosen things to the point where I could create a new site using PowerShell (still no luck using Central Administration). I also (strangely), had to specify an outbound e-mail server first!?!?! ULS Viewer unveiled that mystery, as well as an error attempting to create an account for my logged on user (which obviously already exists) in Active Directory. This error didn’t prevent me from creating the site, but this behaviour confirmed that the site was definitely running in Active Directory Account Creation Mode.

What is Active Directory Account Creation Mode?

Unfortunately, current Microsoft documentation is non-existent as far as I can tell, so I’ll start with the TechNet description from WSS2:

A new feature of Microsoft Windows SharePoint Services is account creation mode for Active Directory directory service. This feature replaces the local account creation feature in SharePoint Team Services 1.0 from Microsoft. Use Active Directory account creation mode when it is necessary to create new user accounts rather than using existing domain accounts. For example, an Internet service provider (ISP) might need the ability to allow SharePoint site owners the capability to create user accounts or invite users to collaborate on a Web site where existing domain accounts for those users do not already exist.

Basically it’s a hosting mode in which SharePoint creates new users in Active Directory and these are the only accounts that can be used in this mode. I can attribute my lack of experience with this mode to my lack of experience with the free versions of SharePoint. Nearly all of my work has been focused on SPS2003, MOSS 2007 and SPS2010. I can’t pin down for certain whether this mode ever existed in the full versions of the product, but according to this article it is now SharePoint Foundation-only and greyed out for SharePoint Server 2010. This TechNet forum post from published SharePoint author and MCC John Ferringer seems to back up that assertion. This post describes the mode well and also appears to answer my next question (my italics, John’s bold):

SharePoint Foundation 2010 (the “free” version of SharePoint that is the successor to Windows SharePoint Services, or WSS) does have something called “Active Directory Account Creation” mode available, which functions much like what you saw in WSS v2. Accounts are first created in SharePoint, and then added to an Organizational Unit in Active Directory. The problem is that this mode is only available at the time you install SharePoint, (its an option off the Advanced Settings button) and you can’t change that configuration setting after the fact. Additionally, you can’t use existing AD accounts in that SharePoint farm, you’ll only be able to use accounts that you create through the tool and you can’t give an account an email address that’s already used by another account in AD. So you need to be mindful of those limitations if you chose to use that mode.

I wish I’d seen this post sooner. It would have saved me some time. For what it’s worth, my investigation backs up this description as follows:

• There is no facility to disable this mode through Central Administration.
• The STSADM commands only support identifying whether WSS is in Active Directory Account Creation Mode using stsadm.exe -o getproperty -pn createadaccounts. There is no corresponding setproperty command.
• The PSCONFIG commands only support creating the farm in this mode – there does not appear to be a means of reverting from it. I believe the configdb’s addomain and adorgunit parameters are responsible for enabling this mode (I could be wrong – the documentation is a bit scant), but I can’t find a facility for reverting it.
• PowerShell is now the preferred means of creating the farm and would be the preferred means of enabling this mode using the New-SPConfigurationDatabase command. As far as I can tell the DirectoryDomain and DirectoryOrganizationUnit parameters are responsible for enabling this mode now, although again, the documentation is unclear to me.
• I even tried to make the change through the API with the help of a friendly neighbourhood developer. Things have changed a bit from WSS2 to WSS3/SharePoint Foundation. At any rate, we found the attribute and set it to “false”, but unfortunately this did not rescue my nascent farm.

In short, it looks like there’s no way back.

How Could This Have Happened?

Given that this mode is only supposed to be present in SharePoint Foundation, I’m really at a loss to explain how I activated it on SharePoint Server 2010. I may have fat-fingered one of the New-SPConfigurationDatabase DirectoryDomain or DirectoryOrganizationUnit parameters I suppose, but I would be really surprised if I did that in a way that allowed me to successfully run the command. In the end, I reverted to my pre-installation snapshots and rebuilt my farm. I don’t feel that was such a waste now that I’ve found John Ferringer’s description and realise there never would have been a way back, but if you find this post through similar folly, hopefully you won’t waste any time trying to revert the farm like I did.

Interestingly, while researching this topic, I stumbled across a TechNet forum post from Andrew Milsark of FPWeb stating that they’ve, “moved away from using AD Account creation mode all together”.

Context

A couple of months ago I was building a client’s production farm. It was a pretty straight-forward architecture with few unusual requirements. I’d successfully provisioned everything and was deploying the PDF iFilter as one of my last steps. When I ran a test crawl to see if it could pick up the contents of PDF documents, I was surprised to find the Local SharePoint sites Scope contained zero items, even though the crawl successfully gathered 459 items. To add to my confusion, the People scope was fully populated. I verified that the scope didn’t need to be updated, then launched ULS Viewer. While reading the trace logs in real time, I re-ran a full crawl and spotted this clue (my bold):

AuthzInitializeContextFromSid failed with ERROR_ACCESS_DENIED. This error indicates that the account under which this process is executing may not have read access to the tokenGroupsGlobalAndUniversal attribute on the querying user’s Active Directory object. Query results which require non-Claims Windows authorization will not be returned to this querying user.

Investigation

This error message reveals quite a bit. We know the error occurs in a w3wp.exe process associated with SharePoint Server Search’s, “Query Processor”, and that the application pool identity of this process doesn’t have read access to the tokenGroupsGlobalAndUniversal attribute in Active Directory. This tells us the error is occurring on the SharePoint Search Service Application pool’s identity, rather than on the Search Service (which is not a w3wp). After searching for a bit I found a few useful posts/articles, but what really helped me was Russ Maxwell’s article, which I linked to at the top of this post.

I suspect that in his testing, Russ found different scenarios where Pre-Windows 2000 Compatibility Access rights needed to be granted to the Search service account, but in my case these rights didn’t help. His error and his explanation of the problem are different. I don’t want to make too much of this, since his post was circa Beta, but it’s worth noting there may be multiple issues with these rights and Search. In our case, we tried to grant rights to the Search Service account but the error persisted until we added the Search Service Application Pool Identity account to this group. In actuality, we identified these same errors on the farm account initially as well, but granting these rights to the farm account didn’t solve the problem.

I should also note for completeness, that there were Security event 4625 Logon Failure errors accompanying the ULS log entries until we granted access to the Search Service Application Pool Identity account, at which point these events were replaced by 4624 Success events.

After running one more Full Crawl I confirmed that the ULS errors were also gone. It’s reasonable to infer from these new ULS events that when PluggableSecurityTrimmerManager is selecting, “workid from scope()”, it needs these Pre-Windows 2000 Compatibility Access permissions in a Windows Server 2000 or Windows Server 2003 domain. Presumably if SIDs can’t be initialised, everything gets security trimmed.

How to use these findings

I’d recommend adding this to the list of permissions you may need to grant in a Windows Server 2000 or Windows Server 2003 domain. This is basically what Russ Maxwell was saying initially, as I read it. In this scenario, I’ve merely spotted a scenario where different rights are required and I can’t shed any light on why this hasn’t been required in every Windows 2000 or Windows 2003 domain I’ve worked in.

If working from a principle of least privileged access, I’d suggest granting these rights as needed during deployment. They shouldn’t need to be granted particularly broadly (unless you’re working with a 1-way trust from a resource domain, which is another story – see the comments in the Russ Maxwell post for an introduction). Alternately, it’s arguable that granting read access to this tokenGroupsGlobalAndUniversal (TGGAU) attribute isn’t opening an enormous hole, but that’s a question for each organisation to answer based on their security models.

A Note on User Profile Pre-Windows 2000 Compatibility Access Rights

While I’m speaking of variance in these permission requirements, I should note that I’ve seen a number of sources including Spencer Harbar, TechNet and this Russ Maxwell article mentioning the need to grant these same Pre-Windows 2000 Compatibility Access rights to the User Profile Synchronisation account, but I haven’t had any problems running without these rights in two different Windows 2000 or 2003 domains.

If the domain controller is running Windows Server 2003, the synchronization account must be a member of the Pre-Windows 2000 Compatible Access built-in group. See Add an account to the Pre-Windows 2000 Compatible Access group for instructions to grant this permission.

Given those sources, I’d suggest you’re probably best off granting the rights, but I haven’t yet been able to validate the need myself. I’d definitely be interested if anyone can shed more light on that topic.

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis

My commentary here assumes some familiarity with these earlier posts. This is new functionality that enables new design options. These options should make SharePoint 2010 on EC2 more appealing for a few specific uses.

VM Import for VMware vCenter

While this first update is quite eye-catching, and will be brilliant for some scenarios, I don’t see it as massively game-changing. I first noticed it in the March 2011 AWS Newsletter, so this is pretty new stuff. I think the blurb speaks for itself:

Amazon EC2 VM Import Adds Connector for VMware vCenter
Amazon EC2 VM Import has released Connector vApp, a virtual appliance that works with VMware vCenter making it easier to import your preexisting virtual machines (VMs) to Amazon EC2. Amazon EC2 customers can use a familiar graphical user interface to select a virtual machine (VM) and specify the AWS Region, Availability Zone, operating system, instance size, security group, and VPC details (if desired) into which the VM should be imported. Once the VM has been imported, you can launch it as an instance from the AWS Management Console. For more information about this feature, see the Amazon EC2 User Guide.

I have yet to put this to the test. There may be big issues with this for all I know, but I figured it was worth spreading the word, as many people seem to be evaluating SharePoint 2010 in the cloud at the moment.

One of the “common uses” listed on the Amazon VM Import page linked above is quite clever I think:

Create a Disaster Recovery Repository for your VM images
Import your on-premise VM images to Amazon EC2 for backup and disaster recovery contingencies. Store the imported images as Elastic Block Store-backed AMIs so they’re ready to launch in Amazon EC2 when you need them. You pay no Amazon EC2 usage charges until you need to launch the instances.

This is probably the most compelling reason to consider this new functionality.

Public Addressing and the Virtual Private Cloud

The next update addresses what I saw as the major problem with Amazon EC2 for SharePoint 2010. Namely, that the Virtual Private Cloud only worked with stretched VPNs to it. There was no public addressing. This is no longer the case, as detailed in another AWS mail-out from a couple of days ago (my bold below):

We are excited to announce greatly expanded functionality of Amazon Virtual Private Cloud (Amazon VPC) that opens up the virtual networking capabilities of Amazon VPC to a much broader set of use cases. Before today, you could provision a private, isolated section of the AWS cloud and launch AWS resources into that VPC that were only accessible via an IPsec Virtual Private Network (VPN) connection to your corporate datacenter. With today’s announcement, you no longer need a VPN or existing infrastructure resources in order to leverage Amazon VPC, but can also connect to your VPC directly through the Internet – you define the virtual network that you wish to use.

With this release, you can now define a virtual network topology in the Amazon VPC that closely resembles a traditional network that you might operate in your own datacenter. You have complete control over the virtual networking environment, including selection of IP address range, creation of subnets, and configuration of route tables and network gateways. You can easily customize the network configuration for Amazon VPC, for example creating a public-facing subnet for web servers that has access to the Internet, and placing backend systems such as databases or application servers in a private-facing subnet with no Internet access.

Amazon VPC enables you to leverage multiple layers of security for access to Amazon EC2 instances, including security groups and network access control lists. Additionally, with the expanded Amazon VPC features, you can:

Divide Amazon VPC’s private IP address range into one or more public or private subnets to facilitate running applications and services in Amazon VPC.

Control inbound and outbound access to and from individual subnets using network access control lists.

Attach an Amazon Elastic IP Address to any Amazon VPC instance so it can be reached directly from the Internet.

Store data in Amazon S3 and set permissions so the data can only be accessed from within Amazon VPC.

For more information on Amazon Virtual Private Cloud, visit aws.amazon.com/vpc.

The earlier design of the VPC (or more precisely, the lack of user-controlled NAT) was the major technical shortfall in the Amazon IaaS offering relative to a traditional datacentre. I haven’t yet looked at this in enough detail, as I’m not actively using EC2, so no doubt there are design nuances that will obtain. Also, the VPC and Elastic IP addresses have associated costs and this will add considerable (likely prohibitive) complexity for the network-untrained. Alternately it may require proper administration, and those associated costs.

For my personal use, I see this as a decisive improvement that will make me reconsider EC2 much more closely for testing cross-farm scenarios, perimeter security, proxies and anything that requires a full infrastructure. While much of this may have been achievable without these VPC changes, it would have been massively complex, more costly and less robust.

17-03-2011 update: as my former colleague Glyn Clough rightly points out in the comments here, I forgot to mention that EC2 also supports Windows Server 2008 R2 now, which is sweet!

15-04-2011 update: Paul Culmsee pointed out today that the VPC is still in Beta, which I’ve somehow managed to forget between December and March. This could mean pricing or other changes before RTM. Also note: Paul pointed out that the VPC does not span availability zones, which could be an issue for global organisations.

• I have an intranet and a MySite, each with a default zone used by all users to access the application, in this example https://sp and https://my.
• I want to optimise my crawl performance by crawling over HTTP, in order to remove the encryption/decryption load for each request, so I create a new Alternate Access Mapping zone on HTTP, for each of these web applications. These new URLs are http://s and http://m.
• I also set up HOSTS file entries on my Crawl Component servers so they will be able to crawl these applications locally. I don’t want my crawls to add load to the WFE servers.
• If I don’t want real people to access the site on this zone (and I probably don’t), I don’t set up DNS entries for it. Only Search will be using this zone.
• (Optionally) I create a Web Application User Policy that restricts access to this zone, since it is not running under SSL.
• I reconfigure my Content Sources in the Search Service Application to crawl the new zones, http://s, http://m and sps3://s
  • Note, for an SSL-secured site, the final People Search Connector, “sps3://s” will actually be “sps3s://s” by default, so make sure to get rid of that last “s”.
• I run a full crawl and verify that it completes much faster than it did when I was crawling the site over HTTPS/SPS3S.
• Once my full crawl completes successfully, I verify that All Sites and People search results are returning as https://sp and https://my (the default zones).

This was all working as expected – until Anthony noticed that the Org Browser web part appeared to be broken. On further inspection, we identified that:

• Alternate Access Mappings didn’t appear to be working on this link (it displayed the crawled address).
• All other links on the People Search Results tab pointed at the default zone; they accurately respected Alternate Access Mappings.
• All results for the All Sites tab/scope successfully respected Alternate Access Mappings.

So we have one link to the wrong zone on the Enterprise Search Centre’s out-of-the-box People Search results page (or tab, if you prefer). I took this scenario to my development environment to confirm. Indeed, the behaviour was the same. I’ll illustrate below.

Alternate Access Mappings for my Blank Site

Alternate Access Mappings for my My Site

The updated Content Sources page

The “All Sites” Search Results Page With All Links Mapped

In the following four screen shots of the same People Search results, notice the status bar as I hover over various links on this page (see: bottom left of the browser).

The Correct Link to Me

The Correct Link to “Add as Colleague”

The Correct Refinement Link to “All Matches”

The Incorrect Organisation Chart Link (http://m)

It would be incorrect to say that this link is broken. It works, so long as you have name resolution for it. The link is just pointing at the wrong zone, which may not be in DNS. Further, clicking that link may not be desirable or could disrupt the user experience, if the zone is locked down. In short, we explicitly want to avoid edits from that zone in this case, for a number of reasons, all of which I will gloss over here as my preference for a single zone.

Long story short: this looks like a bug to me. I’ll call it a bug. These links are new to User Profiles in SharePoint 2010, so there’s a likelihood that Alternate Access Mappings were overlooked here.

What to do?

I still want to use Alternate Access Mappings, because the performance overhead of crawling encrypted data is not unsubstantial. This leaves me with an option to escalate this issue through Microsoft Support, which I really don’t have the time to do, or we could probably whip up some script quickly to update the results page and fix these two links up, but that’s not a very elegant solution. This is no longer a pressing need in my case, because this client has opted to delay launch of MySites and People Search for reasons completely off this map, but the bug remains and pertains elsewhere (like in my development environment). I will take this to Microsoft when I can find the time, but that’s not going to happen in the near future. Until then, any other ideas?

17-03-2011 update: David noted in the comments that he’s worked around this by running his sps3:// crawl on a different web application – even creating one with no real content, specifically for this purpose. I haven’t had a chance to test it out yet but it sounds like a great idea to me. See the comment for more detail. Anthony has put this to the test for our client and all is working now. Hooray!

Afterword: Server Name Mappings

Somewhere along the line I’ve got myself in a muddle regarding Server Name Mappings. When we first encountered this problem, I configured both Alternate Access Mappings and Server Name Mappings (with the same mappings). In this case, I don’t believe this has caused any problems, but it’s not necessary and isn’t correct. Alternate Access Mappings should translate search results to the same zone that you’re browsing from, without doing any extra work. Server Name Mappings translate crawled data like file shares to other links that don’t already exist as Alternate Access Mappings. As this Enterprise Search blog post explains, “Although Server Name Mapping and Alternate Access Mapping achieve seemingly similar results, they work independently, addressing different problems, and should not be used together”. This is perhaps the only content I’ve found that clearly explains how to use Server Name Mappings correctly, and is well worth a read.

It’s worth noting that the Server Name Mappings had no impact on the Organisation Browser link either.

As I started preparing for these exams last week, I was under the misapprehension that there was very little in the way of guidance on the Microsoft Learning site. I think I perceived things this way because there wasn’t much to go on when the exams were first launched. But when I checked again last week I was happy to find a link to a learning plan from the Preparation Materials tab of the 70-667 page. Unfortunately, the 70-668 page does not contain a link to a learning plan (it’s listed but unlinked), but it does exist! I just searched for it on the Training Catalogue.

I also had a look at other resources, such as the Accelerated Ideas free practice tests, Paul Grimley’s post and Benjamin Athawes’ very thorough review. While I found each of these resources helpful in different ways, I opted to stick strictly to the learning plan, focusing purely on TechNet for the remainder of my study time. I basically clicked through to every link from the Learning Plans – many of which contain many more links. I’ve just looked at my browser history for the last week. There’s 267 TechNet pages in there. Obviously, this is a huge amount of information to take in (my brain really hurts), but the quality is great and it’s very well targeted for a crash course/review like this.

Probably most importantly, I learned loads! A lot of this content didn’t exist when I was focused on learning SharePoint 2010, so I (re)acquainted myself with lots of topics that I really should have known more about before starting this process. While there are other more exam-specific approaches to preparation, I feel like I got a great tuition out of this process. I’ve been marking many links to revisit when I have more time, I’ve validated some things I was unsure of and I’ve corrected some misunderstandings. All of this is really valuable for improving quality and for gauging what we do and don’t know.

I managed to pull this off with a little over three days of study (for both exams, in total). About a day of that time was spent figuring out where to start, getting distracted, etc., but I was also able to skip quickly through some content that I was already familiar with, so I reckon 3-5 days is probably a decent estimate of how long it will take to prepare from these learning plans. Keep in mind: I’ve been working with SharePoint for seven years, and I’ve been on the Ignite training for SharePoint 2010, so I may be taking some things for granted.

How did I do? I got a 950/1000 on 70-667 and a 930/1000 on 70-668. 700 is required to pass. In retrospect, this may be over-preparation, but I neither know where else I might draw the line of sufficient preparation, nor do I have any regrets about the additional knowledge I’ve acquired. SharePoint is a complicated beast. Exams like these are sometimes the only catalysts for learning broader/deeper than our daily duties allow.

Round I

My difficulties started when I attempted to move a newly-provisioned Query Component to a web front end server. When it failed, I tracked the problem down to missing permissions on C:\Windows\Tasks. At this point I didn’t know why the permissions had been removed and this was actually the first time I’d noted these permission requirements. TechNet suggests WSS_ADMIN_WPG needs Full Control of %WINDIR%\Tasks, but the description of this requirement is “N/A”. Oddly, according to this TechNet article, the WSS_WPG group does not appear to need these same rights, although they are assigned by the SharePoint installation/configuration processes – or at least they are in the environments that I’ve built.

Adding to this confusion, I found this strange ULS event, in which the provisioning process tries to remove WSS_WPG access to %WINDIR%\Tasks and grant R/W access to the Search service account. This is pretty weird! It might explain why the WSS_ADMIN_WPG group needs Full Control rather than just R/W access, but I wouldn’t typically expect SharePoint to be modifying ACLs in the Windows directory.

“Modifying ACL to allow R/W access to ‘C:\Windows\Tasks’ and to remove access for WSS_WPG.”

Back to the provisioning problem at hand, once I added the missing permissions for both the WSS_WPG and WSS_ADMIN_WPG local groups on %WINDIR%\Tasks the provisioning process completed successfully. You can also see that the “Modifying ACL” event directly precedes the failure to start the new Service Instance. While this event helped me track down the problem, and is clearly related to it, unfortunately I need to leave that mystery behind for now, as there are bigger issues to address in this post.

Round II

Later, this client got back in touch and mentioned that their Search Service Application wasn’t working. In this case the Search Administration page was available but all Content Sources, Scopes, Crawl Logs, etc. pages failed with errors on the Admin Component.

Crawl status: The search service is not able to connect to the machine that hosts the administration component. Verify that the administration component <GUID> in search application ‘<Search Service Application name>’ is in a good state and try again.

To cut a long story short, my initial troubleshooting didn’t immediately lead me back to these missing permissions due to a number of other concurrent infrastructure changes which lead me astray. Additionally, when we tried to delete the Search Service Application to recreate it, the deletion failed after removing just one of the Search databases. Eventually we managed to re-provision the Service Application but the topology changes failed again, at which point we identified the missing %WINDIR%\Tasks permissions (again) and granting the missing permissions fixed these problems (almost).

In fact, we also needed to grant missing permissions on \Program Files\Microsoft Office Servers\14.0\Data\Office Server, but I believe that was a one-off related to the failed Search Service Application deletion earlier. One way or the other it doesn’t appear to be a core issue here. However, I should also mention that I suspect the Search Service Application deletion failed because of the missing %WINDIR%\Tasks permissions – although I’m basing this entirely on the fact that the ULS events above suggests that a similar process takes place for deletion, by virtue of the “(un)provisioning” job.

Round III

With Search back up and running, we moved on to other things, but eventually Search started acting up again. Unfortunately I’ve lost track of the visible failure, but the application logs were full of 6398 and 6482 errors (which typically indicate the unavailability of the service rather than the cause). I vaguely recall that we had items in the index but that new crawls were failing to run. At the time, I was most focused on Gatherer Access Denied messages on the Portal_Content Catalog.

Again, to abbreviate other misguided efforts related to on-going infrastructure work, we eventually found out that the permissions on %WINDIR%\Tasks were missing. Obviously, at this point the most reasonable explanation for the change was a Group Policy setting, so we reviewed the event logs in between the last known good crawl and the first crawl failure. I quickly spotted a Group Policy change message. I recommended that we review the Resultant Set of Policy on this server, just to be absolutely certain the Group Policy wasn’t applying permission changes in this location. The client assured me this was very unlikely, because they don’t have an overly restrictive culture, but it turned out this was the one and only file system permission change and it was applied to the Default Domain Security Policy. Presumably the previous Search failures occurred after reboots or some other event that would re-apply this group policy. And presumably all of this strange behaviour can be accounted for by these missing permissions, given that we know they were getting removed and we know that adding them back in fixed the problem.

Conficker

Later that night, curiosity got the better of me. I dug a bit deeper to see if I could identify anything that recommends these permission changes. I found Microsoft Support KB article KB962007, Virus alert about the Win32/Conficker worm. In this article, Microsoft recommends the following mitigation steps to prevent the virus from spreading:

Set the policy to remove write permissions to the %windir%\Tasks folder. This prevents the Conficker malware from creating the Scheduled Tasks that can reinfect the system.To do this, follow these steps:

1. In the same GPO that you created earlier, move to the following folder:

   Computer Configuration\Windows Settings\Security Settings\File System

2. Right-click File System, and then click Add File.
3. In the Add a file or folder dialog box, browse to the %windir%\Tasks folder. Make sure that Tasks is highlighted and listed in the Folder dialog box.
4. Click OK.
5. In the dialog box that opens, click to clear the check boxes for Full Control, Modify, and Write for both Administrators and System.
6. Click OK.
7. In the Add Object dialog box, click Replace existing permissions on all subkeys with inheritable permissions.
8. Click OK.

In effect, this Group Policy removes the special Read/Write permissions assigned to Authenticated Users on the %WINDIR%\Tasks folder by default. Note: it replaces all permissions with those defined in the Group Policy. I suppose the moral of this story is not to apply security settings like this to the Default Domain Security Policy. But fair play to my client for the security diligence in the first place.

Default %WINDIR%\Tasks permissions for Authenticated Users, without the group policy

This issue raises a couple of other questions. What is the best way to handle this for SharePoint servers, given that there are legitimate reasons harden this location? I suppose the best option would be to create another Group Policy for the SharePoint servers OU which will add the local WSS_WPG and WSS_ADMIN_WPG group permissions back on the %WINDIR%\Tasks folder. There will be other options, depending on how your domain/Group Policies are structured, but this illustrates an approach. It would be helpful to understand if the Search account should be added as well, but for now I’m going on what the installer/configuration wizard does rather than what TechNet fails to describe fully.

Next question: why isn’t this issue more common, given that the virus first emerged over two years ago? I suppose the group policy might not have been taken up by many organisations, but it’s more likely that there are further wrinkles I’ve not uncovered. I tried to replicate the problem in my single server + DC development environment, but frustratingly, everything worked fine after applying this group policy. I rebooted and confirmed the permission changes, ran a full crawl, ran a query and reviewed event logs, but all seemed fine. I even re-provisioned my Search Service Application and that succeeded. To be perfectly honest I’m not sure what to make of this. Perhaps this is only an issue once the search topology takes a specific shape? That feels like the most likely explanation. I hope to do more testing on this in future, but for now I wanted to identify a fix that worked for me and which aligns with the settings applied by the SharePoint installer/configuration wizard, should this problem arise for others. I’m not the first person to discover this problem. I think it’s actually been around since MOSS 2007, based on some forum posts, but I haven’t seen it described in relation to this Conficker protection, which hopefully helps make the Group Policy modelling decisions a bit less obscure.

More broadly, I’d be really curious to hear if anyone has information about the mismatch between TechNet and SharePoint default permissions on %WINDIR%\Tasks, and the further mismatch between the “Modify ACL” event, TechNet and the default settings. It may turn out that the WSS_WPG permissions are unnecessary or even undesirable, but given that SharePoint puts them there in the first place, I’m uncomfortable removing them until there’s better information to rely on.

A quick word about the DCOM Permissions

In my last post, I put off a discussion of the security implications of granting the Farm account DCOM Local Activation rights to the Windows Installer Service (in order to clear the DCOM 10016 event log errors). I was worried about this approach, since this DCOM Component opens up the Windows Installer, which represents a different type of security risk than say… IIS WAMREG. Following my last post, Spencer Harbar suggested that these worries were unfounded, or rather, that the risks are acceptable, since it’s only a risk if the Farm account gets compromised. He rightly pointed out that you’d be pretty stuffed at that point anyway. Fair enough. To this end, I’ll join him in not worrying about it.

How to fix it
If you want to clear the DCOM 10016 errors by granting these rights, you need to assign ownership of HKCR\AppId\{000C101C-0000-0000-C000-000000000046} to Administrators, then grant Local Administrators Full Control. Now you’ll be able to grant the DCOM Local Activation rights to the Farm Account on this same {000C101C-0000-0000-C000-000000000046} component.

Despite carrying a lighter weight on my shoulders, I think it might be helpful to review what came out of my testing, as the job may not be detecting everything we’d expect at face value. I’ve also poked a few more holes in the Support response, which was the whole reason I started working on this in the first place.

Testing the Job

In these tests, I’m wilfully trying to do stuff you would never want to do in any farm – just to find out what the job “knows” about. To this end, I’ve tried some pretty foolhardy things like:

• Manually updating DLLs in the GAC.
• Manually updating DLLs in the Program Files directories.
• Manually killing a Cumulative Update installation while it was half-way complete.
• Deleting DLLs from the GAC and the Program Files directories.
• Manually updating registry keys.

Are these the right tests? They certainly aren’t comprehensive. Suffice it to say I’m not the right person to comment on what the Windows Installer might be able to detect. In the process of researching this I’ve already become far more acquainted with Reflector and the Windows Installer than I ever hoped to be. I’ve even found out that there’s a Windows Installer blog and Windows Installer MVPs. Who knew? But are these changes the types of things that could cause disruption in a farm? Probably. And should we understand if the Manage Patch Status page in Central Admin accounts for problems like these? I think so. Thus, this imperfect testing by the wrong person.

Replacing DLLs

In the first two tests below, I copied DLLs out of an installed instance of the December Cumulative Update and replaced the installed June Cumulative Update versions of these DLLs in another machine with these newer copies. The DLLs I was looking at were for Microsoft Excel Services Components and Microsoft InfoPath Forms Services (this is how they are listed on the Manage Patch Status page).

Manually replacing a DLL in the GAC

When I manually deleted my June CU Microsoft.Office.Excel.Server DLL from the GAC using GACUtil (as you shouldn’t do), and replaced it with a newer version from the December CU, I broke my Excel Services Service Application. When I ran the Product Version Job timer job it failed to detect the change (the new version was never reflected in Manage Patch Status). Everything looked exactly as it normally would in the application event log, except for this message immediately after the normal 1015/1035 entries:

The Execute method of job definition Microsoft.SharePoint.Administration.SPProductVersionJobDefinition (ID 9bb9d31b-7c8b-4fd7-b52d-5fec40aa3607) threw an exception. More information is included below.

Failed to call GetTypes on assembly Microsoft.Office.Excel.Server.MossHost, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c. Method ‘IsEditEnabledForCurrentUser’ in type ‘Microsoft.Office.Excel.Server.MossHost.MossHost’ from assembly ‘Microsoft.Office.Excel.Server.MossHost, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c’ does not have an implementation.

This error is informative, and would probably help me track down the issue in due course, so the Product Version Job is earning its keep, but it’s unfortunate that this version change is not displayed in Manage Patch Status in any way. In short: this is a good reason to run the job but it’s also good to know this kind of problem won’t appear in Manage Patch Status.

Manually Replacing a DLL version in the Program Files directories

Next, I tried to manually replace DLLs in the Program Files directories with newer versions. I searched throughout the Hive and the C:\Program Files\Microsoft Office Servers\14.0\ directories for other versions of these files. I was working on the assumption that the version in the GAC would be in use (thanks to Chris O’Brien for this advice), but I wanted to see if the job would successfully spot changes in these Program Files locations, since this is what the Microsoft Support response suggested.

I found the same InfoPath DLL and a differently-named Excel Services DLL in these locations:

• C:\Program Files\Microsoft Office Servers\14.0\Bin\Microsoft.Office.InfoPath.Server.dll
• C:\Program Files\Microsoft Office Servers\14.0\Bin\xlsrv.dll

I ran the Product Version Job after deleting these files and rebooting. Again, the job failed to detect the changes.

What happens with added DCOM Local Activation rights?

If the farm account has DCOM Local Activation rights on the Windows Installer Service, it resolves the DCOM error event log clutter, but these rights don’t impact whether the job can detect these changed DLLs.

Killing an installation part-way through

Next, I rolled back to a stable state and ran the December Cumulative Update against a June Cumulative Update installation. At a random point during the installation I killed the installer (not the Products Configuration Wizard). While the installer was running I wasn’t able to monitor activity in ULS Viewer because SharePoint was being patched. However, I was looking at the dbo.ServerVersionInformation table in SQL Management Studio and I could see new rows with updated versions appearing as it progressed. The Cumulative Update installer was writing to the same table that the Product Version Job updates.

Running the Products Configuration Wizard after fixing the failed installation

Later, I fixed up my December CU installation and ran the Products Configuration Wizard. When it was running, I could see that something very similar to the Product Version Job was logged. The same informational events (1035) appeared successfully in the application event logs, without any DCOM errors or “Failed to Connect to Server” (1015) application event log warnings. Presumably this succeeds (with or without the DCOM rights) because the Setup account that’s running the wizard is a local admin and therefor already has the DCOM Local Activation rights. However, I’m not sure what’s gained by updating Manage Patch Status at this point, since the dbo.ServerVersionInformation table was already updated by the installer. I won’t dwell on that thought too much though, since there may be a very good reason for the update at this time.

For those who are interested in the workings of this update, it’s worth noting that the Products Configuration Wizard appears to use the Microsoft.SharePoint.Administration.SPServerProductInfo.UpdateProductInfoInDatabase(Guid serverGuid) method. It effectively calls the same thing as the Product Version Job timer job, if I’m reading all of this right. A fuller glimpse of the ULS logs looks like this:

Updating SPPersistedObject SPServer Name=SPSQL. Version: 120278 Ensure: False, HashCode: 2459215, Id: 20c667df-1bc3-486b-869c-a3ba40f83af5, Stack:
at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdate()
at Microsoft.SharePoint.Administration.SPServerProductInfo.UpdateProductInfoInDatabase(Guid serverGuid)
at Microsoft.SharePoint.PostSetupConfiguration.FinalizeTask.Run()
at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()
at System.Threading.ExecutionContext.runTryCode(Object userData)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()

It’s also worth noting that this log entry correlates with the MsiInstaller 1035 success events in the application event logs that I mentioned above.

What about deleting DLLs?

While investigating this, I ran all of this by my colleague Jalil Sear. He came up with an interesting idea: that I shouldn’t just update the DLLs, but I should try to delete them altogether. So I deleted Microsoft.Office.Excel.Server and Microsoft.Office.Infopath.Server from the registry and the GAC and reset IIS. I re-ran the Product Version Job and it completed normally, with and without DCOM Local Activation rights. Nothing was detected, although my entire Manage Service Applications page was annihilated. Again, we might have expected this to be reported in Manage Patch Status.

Summary of Test Results

• The Product Version Job reports “Success” in the Timer Job Status, regardless of all of these considerations. It may fail for other reasons, but all of these issues obtain when the job reports a successful status. In other words, the job reports “success” with or without DCOM rights.
  
• It’s not clear to what extent the Product Version Job can account for problems while the installer runs, because the installer already makes updates to the dbo.ServerVersionInformation table as it goes.
  
  • One might reasonably wonder what would happen to whatever was being updated while the installer failed. Obviously it’s hard to make broad statements about that when we don’t know at which precise point it failed, but in any case the remedial action will be to run the installer again - potentially after fixing something else. One way or the other, if you have this problem, I don’t see how the timer job is going to help because it’s unlikely it will be able to run against this server until the installation is fixed.
• It’s also not clear to what extent the Product Version Job can account for issues that occur while the Products Configuration Wizard is running – effectively for the same reasons as above. If you have a problem with that wizard, the remedial action will be to fix the problem and run the wizard again.
• Manage Patch Status doesn’t seem to account for other issues in the GAC or the Program Files directories, such as manual changes to DLLs. Presumably this is because these actions have been taken without using the Windows Installer Service.
  • Obviously, if you’re running an environment where these sorts of changes are routinely possible, then this job is a lesser concern than Change Management processes that might prevent these things from happening in the first place, but it’s worth knowing that the job did not detect these changes in my tests.
• It’s not clear in which cases the Product Version Job is useful for recording the difference between product versions on different servers, since the installer should have already updated the dbo.ServerVersionInformation table.
  
  • One example where the job might be useful is the case where a server is restored to a pre-upgrade state. However, it’s likely that this restore operation will prompt some other remedy, like reverting all of the other servers in the farm or upgrading this server again. So the usefulness feels limited to me. Still, this is probably sufficient reason to run the job absent any other considerations.
• The Manage Patch Status page is still useful for tracking differences across servers where the servers are legitimately running at different patch levels, although typically that’s not a state you’d want to run in for long.
  

Putting this information to use

I wouldn’t suggest reading this as the full story, since I only ran these against a single SQL/SharePoint box. At a minimum the Product Version Job can detect product version mismatches when a server is restored, and servers in long-term mismatched states. As a plus, it will throw an error in your application logs to let you know if there’s something wrong with the DLL that it expects in the GAC. Unfortunately, that isn’t reported to Manage Patch Status. In any case, as teams/farms increase in size this job becomes more useful for shared understanding.

At the end of this review, I think the important thing is to recognise the limits of the data in Manage Patch Status. It’s not going to be bullet-proof. For any actions taken with the Windows Installer, this data should be pretty reliable, since it’s updated during install, with the Products Configuration Wizard and with the Product Version Job. For anything else - who knows? It doesn’t appear to have been designed for that, and I have no idea what a SharePoint timer job would look like that could offer these kinds of assurances. Presumably it would have to be a management agent of some sort. At that point you’re in to Configuration or Operations Management territory and we already have different tools for that. Come to think of it, if you really want to know, “the install state of the machine“, that’s probably what you’re really looking for. But if you want to know the current versions of successfully-installed SharePoint Products on all servers in your farm, then Manage Patch Status should be accurate in most cases, because of the Product Version Job.

This is a fairly lengthy consideration, but I think it’s necessary to cover these details because the information in the Managed Patch Status (AKA Check Product and Patch Installation Status) page in Central Administration may not be revealing what we’d reasonably infer.

In this post and the posts to follow, I’ll cover a few things:

• Why I think granting Local Activation rights to the Windows Installer Service puts a dent in the least-privileged model.
• What this DCOM error means to the reliability of data displayed in the new Manage Patch Status page in SharePoint 2010 Central Administration.
• What the job does and doesn’t do, with or without rights to launch the Windows Installer Service.
• Considerations for disabling the Product Version Job timer job.

The Problems

I believe most people will come to this problem in the way that I have, which I’ve seen repeated on many TechNet fora since then. People want to know why they are getting inundated with approximately 100 DCOM 10016 System event log errors and twice that many MsiInstaller Application event log warnings and informational events nightly, at around 00:52. The exact number of messages will vary based on the SharePoint products installed in the farm, including related products such as Project Server, Office Web Apps, FAST Search, etc. For a more detailed review of these events and how they can be identified, please refer to my original post.

Additionally, we have a nightly timer job which seems to be failing, per these DCOM errors. The job itself claims to check, “the install state of the machine and puts that data into the database”. This is rather vague. As of August this is what I understood:

• The timer job appeared to fail to use the Windows Installer Service to perform a check of installed SharePoint products.
• I didn’t know anything about how that check happened or how the data was used afterwards.
• I didn’t know if the event log messages were ephemeral (annoying only because they generate clutter), as they are for the IIS WAMREG DCOM 10016 errors.
• I felt it would be bad to grant rights to launch the Windows Installer Service to the farm account in an otherwise-least-privileged configuration (where the Farm account does not already have local administration rights).

In this post I want to dwell on the inner workings of the  job itself, and then come back to the implications for our event logs, permissions and job scheduling.

Inside the Job

In order to find out how the job works, I had to crack it open in .NET Reflector and SQL Management Studio. I need to disclaim this post, because I’m not a developer, and to be perfectly honest I’m in a bit over my head with Reflector, but I was prompted to investigate in this way based on the apparent misinformation in one of the TechNet threads I mentioned above. Geoff Belair went to considerable lengths to work through this topic with Microsoft support, but from what I can tell, there are a number of mistakes in the answer he received. It suggests the wrong database gets updated and is a fairly inaccurate description of what this job does, by my reading of the following clues.

(In)validating the Microsoft Support Explanation

It’s unfair to take a Microsoft Support e-mail which has been re-posted on the web as authoritative, but this was the closest thing to official information I’ve found, other than the brief words about this job on TechNet and MSDN. The key bit of that reply that I wanted to immediately verify was this:

The Timer Job “Product Version Job” runs every night at 12:52 A.M and analyze which are the dlls are updated, once it get the information then it’s put the updated version data on to Content Database “dbo.version” table.

So I took a look at the dbo.Versions table in the Central Admin Configuration database (never do this in production, of course).

What caught my eye was that there was no product information in this table whatsoever. I knew that the job was checking for the state of individual products based on the MsiInstaller informational events in the Application logs. So I poked around a little more and found what I was expecting in the dbo.ServerVersionInformation table:

Having looked at this data, I realised it was pretty familiar. I went back to Central Administration, looked in the Upgrade and Migration section and clicked on Check Product and Patch Installation Status, which took me to this Manage Patch Status page. The key thing to note is that the version numbers and the Patch Status columns match the data on the page below precisely. I’ve actually manually updated that data just to give it a sneaky check, and this page is definitely pulling it in from that source. You’d never do this on a real system, however. I wouldn’t even do it without having a recent snapshot for my development environment.

At this point I was pretty confident the timer job was trying to update this table, but I wanted to get a bit better assurance before testing the job in anger. I also wanted to understand how the Windows Installer Service gets involved, as this activity seems to take place outside the ULS logs.

Analysing the job in .NET Reflector

Cracking open ULS Viewer while running the timer job, you immediately see the fourth event in my screenshot below. Job-admin-product-version calls SPProductVersionJobDefinition.

This is where I opened Reflector. I started with Microsoft.SharePoint.dll and drilled down to Microsoft.SharePoint.Administration.SPProductVersionJobDefinition, which executes SPServerProductInfo.UpdateProductInfoInDatabase(Server.Local.Id);

SPServerProductInfo calls the GetMsiData method, which works with a number of SPMsi methods (SPMsi.GetPropertyUsingProductCode, SPMsi.MsiEnumPatchesEx, SPMsi.MsiGetPatchInfoEx, SPMsi.SPMsiSafeHandle, SPMsi.MsiOpenDatabase, SPMsi.MsiDatabaseQuery). Further down, SPProductVersionRow is clearly collecting the same data as the columns of the SQL dbo.ServerVersionInfromation table I examined earlier. If interested in these workings, I’d recommend perusing it with Reflector at a more leisurely pace than this.

Note: all of the SPMsi.Msi* methods are using msi.dll, which is the Windows Installer.

The Product Version Job’s use of the Windows Installer

From here, I could explore the workings of the Windows Installer in finer detail – but for the purposes of our SharePoint knowledge, all that’s really important to know is that the timer job is using the Windows Installer’s own methods to query the installed product versions on the servers. As I understand it, the Windows Installer typically stores this data in the registry, at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\. That’s an oversimplification, but stick with me for now.

We can see a number of SharePoint products in this location. They are all keys beginning with “90140000″. Taking a look at the data in these keys, it’s pretty clear that it aligns with the data that’s written to SQL’s dbo.ServerVersionInformation table (down to the registry key value in the “Patchable Unit” column). Additionally, these are all the same products that are identified in our Application event log messages. You can even see the patched products have a longer key, with a suffix that looks something like “_Office14.OSERVER_{48017E90-141F-4948-A576-F4B9B6284B70}”.

Perhaps most importantly, the ProductVersion Property of the Windows Installer is what defines the four “version” values (including “DisplayVersion”) of the Uninstall keys above. This is the key information that the Product Version Job is after, and the name of this timer job feels like an even better fit in this context.

While unravelling the job in this way has given me a fair amount of confidence about how SharePoint retrieves this information, there are still a number of issues to consider. For starters, I suspect people look at Manage Patch Status data and feel pretty confident about that representation of the installation state of their servers. Being a fairly skeptical type, I suspected that the Windows Installer’s “record keeping” would be good up to a point, but no further, so I put on my demolition hat and started breaking stuff, in an effort to place that point. In my next post I’ll review those test results, then consider the implications for DCOM rights to the Windows Installer Service and the timer job scheduling options.

UPDATE 19/2/2011: I got a comment from Spencer Harbar today (below) noting that restarting the SPTimer service is sufficient after temporarily adding the farm account as local admin. The reboot isn’t necessary to acquire the new rights although in my test I did need to reboot after running the installer.

Other bad stuff about it:

• It took forever to install and it required a reboot afterwards.
• The Products Configuration Wizard gets to step 9 of 9 quickly, then gives a % complete status. Mine was running at 10% with no new updates for about 20 minutes before it failed. After it failed, all of the installation log errors were related to the User Profile Service Application, so I rebooted and ran the Products Configuration Wizard again. This time it completed successfully fairly quickly but the Patch Status and Upgrade page in Central Admin didn’t get updated for this next run. Not sure what to make of that, but everything seemed fine except for the User Profile Service Application when I logged back on.
• I have some new FIM-related application event log errors regarding FIM workflows that I haven’t had a chance to look in to, but a Full Sync seemed to be working so they may not be important.

What I’d recommend:

• Run the installer and add the Farm account as local admin before rebooting.
  • Note: per Spence’s comment, this isn’t necessary. Restarting the SPTimer service is sufficient. The installer may prompt for a reboot though.
• Run the Products Configuration Wizard, restart the User Profile Synchronisation Service (wait for it) and then remove the Farm account as local admin after it restarts successfully.
• It’s probably still not worth installing this unless you know you need it.

Update: for a more detailed consideration of these topics, check Chandima’s post, which walks through all of this step-by-step and links to much of the supporting guidance than I’ve trampled over here.

As you will have surmised from the title, we were deploying the Office Web Apps. In this case they were being installed with a new SharePoint Server 2010 Standard farm. This is a list of considerations that cropped up during the engagement and a few other bits I’ve picked up since RTM:

• WAN Acceleration: The Office Web Apps help performance over the WAN. This hadn’t occurred to me until I read the recently-released TechNet guidance on geographically-dispersed environments, but this all makes sense, because the documents load progressively. This is explained in more detail in the article. It’s worth keeping this in mind as an aid for global deployments and worth further taking note of the licensing concerns below if the Office Web Apps will be deployed for this reason.
• Licensing: The Office Web Apps license model is based on the volume license for the Office 2010 client. These licenses are supplementary to any SharePoint license concerns, although you might choose to install both in the same farm. This Infoworld article explains it better than anything else I’ve read. The major implications of this are:
  • Assuming this Infoworld article is right, and if the farm will be accessed by external users, they will not be covered by a SharePoint FIS license. All of those users will need to have an Office 2010 volume license (not a home license) in order to access these documents in the browser.
    • This license could be provided by the consuming business or whoever, but it would need to be in place. How this would be monitored, what is and isn’t acceptable use and how licensing audits would work in this scenario are all issues that would need to be inspected in some detail with a licensing specialist.
    • Presumably you could deploy a web application that’s disconnected from the Office Web Apps service applications in order to maintain compliance, but then you negate the WAN optimisation benefits mentioned above, and this clearly has broad IA implications. I’d also recommend confirming the legitimacy of this approach for compliance.
  • On the other hand, internal users are covered by an Office 2010 volume license if it has been purchased, even if the software isn’t installed on their machines. You might find this is the case with enterprise agreements. This means internal users can start to take advantage of the Office Web Apps even if an Office 2010 upgrade is months or years in the future.
  • Talk to a licensing specialist to confirm all of this as it pertains to your deployments, as there are surely other wrinkles I’m not covering and I’m basing this on an article written by a journalist rather than Microsoft or a licensing specialist. The only reason I’m referencing that article is that it’s the clearest explanation I can find.
• Install Media: Once you’ve got the licenses in place, you may have some trouble finding the install media. Even though the media is for 64-bit systems, it’s downloaded from 32-bit downloads on the volume license site.
• Caching: The Office Web Apps Cache is a site collection that should be moved to it’s own database for each web application (in most cases), as the default cache size is 100GB. One way or the other, it’s a different type of content and it makes sense not to clutter actual content databases with this cache data. There are probably different database backup/restore requirements as well. If the existence of this site collection is news to you, read those links above before going any further.
• Topology: Make sure to plan your topology for the Office Web Apps. You will be adding three new service applications and they will tax your system differently based on how they are used. This is a big topic and the planning guidance is fantastic, so I’ll merely point out the Estimate performance and capacity requirements for Office Web Apps document and leave it at that.
• Database Permissions and Application Pools: The Office Web Apps have some quite unexpected database permission requirements. From the Deploy Office Web Apps TechNet guidance:

  Note: You can choose to create a new application pool to be used with a service application. When creating a new application pool, you can specify the security account used by the application pool to be a predefined Network Service account, or you can specify a managed account. The account must have read\write privileges for the SPContent database and SPConfig database. For more information about services account permissions in SharePoint, see Account permissions and security settings (SharePoint Server 2010).
  
  This blows a few holes in the least-privileged model. In this case, we chose to create a new application pool for the Office Web Apps service applications and ran it under the farm account, since the permission requirements are so pervasive. This could be its own separate identity, but the important thing for me is that using the farm account for these services contains the extensive privileges more than granting these wider permissions to the Service Applications application pool would. One way or the other, we’re looking at a different application pool for security reasons. And if anyone’s curious if this is really necessary, it is. You will be unable to actually use the Office Web Apps without these database permissions in an otherwise-least-privileged configuration.

There are a number of other topics to consider before deploying the Office Web Apps, such as the default open behaviour, Office version requirements, browser support and other topics from the solutions world that I daren’t venture in to, but hopefully these infrastructure considerations won’t be overlooked when focusing on those issues.

• SharePoint Development Productivity and Virtualisation Technologies
• SharePoint 2010 Development Environment Performance Tests

I’ve said my preamble in those posts, so I’ll cut to the chase here.

High-Level Summary of Findings

• Disk performance and bus speed did not prove to be significant factors in these results (except for virtual machine start-up times). Obviously there are fundamental differences about SSD (yet untested) that may skew this picture, but I will be surprised to see big differences. If we’ve got these tests right, and they are actually representative of the tasks that slow down development, then we would expect to see wider variance across bus or disk speeds. We don’t.
  • This assumes the disk is relatively uncontended. Virtual machine performance degrades in every type of test while large file operations are running concurrently on the same disk. This could be copying an ISO, importing or exporting a virtual machine or any other sustained large file operations.
    • At a minimum, this is certainly an argument for running VMs on their own spindle, whether it’s over USB, eSATA or SATA. This may be an area where SSD shines.
  • These disk performance figures can be found towards the bottom of this post. Desktop performance was nearly identical running on USB2 at 5400 RPM versus a RAID0 stripe or a RAID1 array on 7200 RPM disks. Laptop performance was also nearly identical over USB2 5400 RPM versus eSATA 7200 RPM.
• Hyper-V performance has been poor on all laptops with i-Series CPUs. This is more pronounced in some areas than others. Our three-year-old model with a Core 2 Duo actually outperforms the new i7 in some cases. When these results are added to known driver issues with Hyper-V on many newer laptop GPUs, we’re looking at a configuration that’s unfit for SharePoint 2010 development.
• VMWare Workstation outperforms Hyper-V on laptops by significant margins in most areas. The exceptions to this are start-up time and performance during the first 10-30 minutes of use (I believe VMWare is ballooning during this time). After that, VMWare Workstation is faster than Hyper-V in every type of test.
  • As a long-time advocate of Hyper-V despite usability deficiencies, I was probably more surprised by the significance of these differences than anyone. I wrongly assumed that Type-I hypervisors would outperform Type-II in nearly every way. While that may hold true on server class hardware, it doesn’t hold true here. I’m a convert.
• While less pronounced, these same findings hold true on the desktop.
  • Desktop performance is very quick on VMWare Workstation, considerably out-performing even Amazon EC2.
  • We can realise significant productivity gains by moving all users who are primarily office-based to a desktop + VMWare Workstation configuration from laptop + Hyper-V, at a fairly small cost (probably half the cost of EC2 over three years – see my recent posts on EC2 for more information).
  • Desktop performance on Hyper-V, while notably slower than VMWare Workstation, is generally faster than VMWare Workstation on the i7 laptop.
• Laptop performance is significantly improved on our current model with VMWare Workstation. These improvements are also realised on the newer model laptop, but the performance delta between the two physical systems is not so significant that it’s compelling to move to a low speed i7 from a reasonable speed Core 2 Duo.
  • The total times for the “End-to-end site creation to debugging tests” were two and a half minutes faster with VMWare Workstation compared to Hyper-V on the Dell XPS M1330. Moving from Hyper-V to VMWare Workstation for laptop users is now an obvious choice.
  • The benefit of spending on i7 processors is in doubt. We are seeing very minor performance penalties when adding more than two CPUs in both VMWare Workstation and Hyper-V for most tests. There were also very minor improvements for some tasks, but on the whole there does not appear to be a measurable benefit. This might vary if the host OS is doing a great deal with the CPU, but that is liable to cause other contention issues than just in the CPU (on a laptop).
  • The only tasks that appeared to use all 8 cores in a SharePoint VM were:
    • Retract/Deploy of a solution (but only very briefly)
    • Create web app, or Create site collection (but at low percentages)
    • Rebuild with Code Analysis (but not fully)
  • We will be running future tests on i5 processors at higher clock speeds to see how these models perform relative to the 1.6 GHz i7.
• The User Profile Service Connection doubles first page load times after an IISRESET in all test cases. I consider this a full validation of these preliminary findings.

Snapshot of key data

The Data

How to read the data:

• Hardware: the physical laptop or desktop model (or Amazon’s EC2)
• Virtualisation: “Hyper-V” is short-hand for the Hyper-V role in Windows Server 2008 R2. “VMWare 7.1.2″ is short-hand for VMWare Workstation.
• #CPU: the number of physical CPU presented to the guest operating systems. Multiple logical cores were only tested in the 4×2 results below.
• Disk: the physical disk configuration where the virtual hard drives are running.
• RAM: the amount of RAM running inside the SharePoint Server 2010 VM. The Amazon EC2 instances were “large instances” but the domain controller was running locally.
• Test: The tests have been described in more detail in my last post.
• Result 1, 2, 3: Each test was carried out three times. The far-right column, Average Result, is an average of the three.
• The Two “Average Load…” rows are an average per-result of the three rows above them. These are tests built on SharePoint 2010 default site templates, which anyone should be able to replicate.
• The “Total create to debug time” row is a sum of the five rows above it.
• All results are in seconds. In cell G21 below, 524 seconds = 9 minutes and 2 seconds.
• For more information on the tests and the testing methodology, see my last post.

Hyper-V versus VMWare tests, all other things being equal

Dell XPS M1330, running Hyper-V

Dell Studio XPS 1645 laptop, running Hyper-V

ASUS V7-P7H55E desktop, running Hyper-V
Note: these Hyper-V tests were accidentally carried out while the VM was running on a RAID 0 stripe rather than on the System disk, so this is not apples and apples, but later disk tests on VMWare Workstation indicated that this shouldn’t make much of a difference, so I’ve left these results in, with this comment.

Dell XPS M1330, running VMWare Workstation

Dell Studio XPS 1645 laptop, running VMWare Workstation

ASUS V7-P7H55E desktop, running VMWare Workstation

VMWare Workstation i7 tests with 4 or 8 cores

Dell Studio XPS 1645 laptop, running VMWare Workstation with 4 CPU

ASUS V7-P7H55E desktop, running VMWare Workstation with 4 CPU

Dell Studio XPS 1645 laptop, running VMWare Workstation with 4 CPU, 2 Cores Each

ASUS V7-P7H55E desktop, running VMWare Workstation with 4 CPU, 2 Cores Each

Amazon EC2 Results

Notes:

• Times were much slower one day than others. This hasn’t been measured over time, but it’s worth keeping in mind. Other EC2 users reported similar problems on the same day.
• Also note: a couple of rows of test data (245 and 248) have been accidentally deleted, but the results were not unexpected in any way.
• Row 263 has no data because measuring time to desktop with EC2 would be too imprecise. It would normally be available within five minutes from start, for reference.

Disk Tests on VMWare Workstation with two cores

The format of these tests change slightly, as I am grouping all disk permutations for the Dell Studio XPS 1645 together, then moving on to the ASUS V7-P7H55E desktop. I grouped them this way because the tests were fundamentally different for laptops and desktops. I did not get the time to repeat the laptop tests on the Dell XPS M1330.

Dell Studio XPS 1645 laptop with VM running on 5400 RPM USB2

Dell Studio XPS 1645 laptop with VM running on 7200 RPM eSATA

ASUS V7-P7H55E desktop with VM running on 5400 RPM USB2

ASUS V7-P7H55E desktop with VM running on a 2nd set of RAID 0 spindles

ASUS V7-P7H55E desktop with VM running on a 2nd set of RAID 1 spindles

…and with that, I’ll let you draw your own conclusions. Should anyone want to contribute supplementary test data in the comments here, or carry out further tests (perhaps with SSD), I would love to see the results. As I mentioned in the last post, there’s still more testing to do.

Update 08 June 2011:SharePoint 2010 Development Environment Performance: SSD, i5 vs. i7, WEI and Sandy Bridge

The Tests

The 21 tests that we settled on were the result of discussions with a number of the core developers, consultants and architects at Content and Code, plus a few tests that I threw in to confirm/disconfirm some of my suppositions, such as the impact of the User Profile Service Connection on first page load time. All 21 tests were run three times for each permutation of hardware candidate and virtualisation technology. We also tested on Amazon EC2. I will discuss the testing process in more detail in a moment.

These tests have been selected for a few reasons:

• They are tests that anyone can run, including Visual-Studio-allergic types like myself.
• They re-enact real-world productivity loss. All tests needed to be significant on our current system or they were thrown out.
• They needed to account for tasks that impact non-developers as well as people that have their head down in code 40 hours/week.
• They needed to be examples of tests that would stress systems in different ways.

First page load tests
These tests were designed to examine what, if any impact different sets of features, functionality and structure might have on first page load times after the application pool is recycled or IIS is reset (while gathering a large set of data to make comparisons across systems). I also wanted to fully validate my preliminary findings about the User Profile Service Connection.

I ran these tests against NTLM-authenticated web applications with the following root site collections:

• Central Administration
• Blank Site
• MySite
• Blank Site, with no User Profile Service Connection
• The Content and Code website solution (structure, without content)
• A custom intranet solution (structure, without content)

All of these first page load tests were repeated for application pool recycles and IIS resets.

End-to-end site creation to debugging tests
I hope these tests are fairly self-explanatory. I used the Content and Code website solution because it’s a public site that people can examine if they want to understand more about the structure of the solution and the scope of customisation tested here.

1. Create new NTLM-authenticated web application from the GUI
2. Create new Publishing Portal Site Collection from the GUI, at the root of the new web application
3. Deploy Content and Code website solution from Visual Studio
4. Delete the publishing site collection (this was a necessary step, but not a test that I timed)
5. Create Content and Code website (structure, without content) from the GUI
6. Debug Content and Code website solution in Visual Studio

Core development tests
These tests were added to account for pure development activity for large projects with lots of dependencies. We turned Code Analysis on for the first test because this is a feature that’s very useful but taxes systems pretty heavily. The code deployment times were all fairly small relative to other tests here, but we need to keep in mind that this could be repeated literally hundreds of times per-day. Note: full deployment is accounted for above in the end-to-end test.

• Rebuild Large Project w/Code Analysis
• Deploy Large Project to GAC/BIN

Disk/IO tests
These tests were thrown in because they have an impact on productivity even if they aren’t particularly routine. For the first test I measured the time from turning on the VM until the desktop rendered after logging on. The second test doesn’t really meet the “real world” criteria I name above, but it is a task that can be a productivity barrier in some cases.

• Time to desktop
• Run full crawl (three web apps, no content)

The Testing

The testing process was entirely subject to personal fallibility, as I carried these tests out myself using fairly imprecise methods like a browser-based stopwatch running on my host system (I made sure not to time things inside the guest, where time can slip occasionally). I also went to great lengths to carry out these tests when the systems were performing optimally; I would run through all of the tests once before recording the first set of results. I felt this approach was the best way to discount random variance. The test results were largely very consistent, so I believe these efforts paid off. Obviously the down-side to testing in this manner is that real work is not carried out in a vacuum, but I don’t see any other way to come up with repeatable tests aside from measures like these. It’s what works for science, after all.

The Virtualisation Technologies

As I mentioned in my last post, I chose to limit the virtualisation technologies to a single technology from each of the types I described. I had to postpone testing against “local systems” due to time pressures. It was the option that fell off because we are unlikely to ditch virtualisation any time soon. It works well for us.

To reiterate here, the candidate technologies were VMWare Workstation 7.1, the Hyper-V role in Windows Server 2008 R2 and Amazon’ s EC2 IaaS offering (a Red Hat implementation of the Xen hypervisor). Again, there’s background for all of this in my last post.

What About the Server Room?

One thing I haven’t discussed in any detail so far is VDI or Remote Desktop services. I briefly touched on shared development environments, but I’ve not talked about hosted, individualised development environments. The reason we ruled this out is cost. While this would probably be the best-performing option, all other things being equal, the costs associated with providing this level of performance in the server room would be pretty enormous. For our purposes we might have exceeded power, cooling and weight limitations before we considered the costs of new blade centres and SANs. These costs would probably be even greater in the datacentre. In short, the same criticism applies to individualised hosted development environments as to shared environments: redundancy and resilience at this level is overkill given the associated costs. The data is not critical and anything that needs to be backed up can be stored elsewhere (like TFS).

Basically, people opt for VDI or Remote Desktop services because a mass of underutilised desktop systems can be heavily consolidated. These systems are not underutilised.

The Hardware Candidates

Dell XPS M1330
This is our current laptop model, upgraded with a 320GB 7200 RPM local hard drive and 8GB RAM. One of the serious options we’re considering is a laptop refresh, due to the age and fail rate of the graphics cards and motherboards on these models.

Dell Studio XPS 1645
This was the least expensive decent i7 laptop I could find for testing purposes, and a leading candidate as a replacement laptop. With an £833 (ex-VAT) starting price it could be bumped up to 8GB RAM for a little over £100 more via Crucial. It’s a very heavy laptop and the glossy shell does it no favours, picking up fingerprints within seconds of use. However, it comes wth a 1.6 GHz i7 processor, 500GB 7200 RPM disk standard, eSATA port and HDMI. No USB3. Basically, nothing here was an absolute deal-breaker for us if performance was good.

ASUS V6-P7H55E
This is a barebones system with the following configuration/cost (as priced at scan.co.uk):

• ASUS V6-P7H55E barebones System = £121.67
• Intel i7 870 (8M Cache, 2.93 GHz) = £217.57
• 4GB Corsair XMS3 DDR3 PC3-10666 (1333) Dual Channel – 4x£56.59 = £226.36
• 1TB Seagate Barracuda SATA 3Gb/s, 7200rpm, 32MB Cache, 8.5 ms, NCQ – 3x£41.94 = £125.82
• Adaptec 1220SA PCI-E RAID Card = £46.40
• ASUS 512MB GeForce G 210 DDR2 NVIDIA Graphics Card = £27.71
• Total = £768.58 (VAT-inclusive)

This system is configured with three internal 1TB hard drives and 16GB RAM. We needed to purchase the RAID card because the motherboard does not have an on-board RAID controller. The graphics card was necessary because there are no integrated graphics on desktop i7 processors (although there are for some i3 and i5 models). The disk configuration was variable, as this was one of the test scenarios. The assumption going in was that two disks would be configured in a RAID 0 stripe or a RAID 1 array, depending on performance outcomes. We would only stripe the disks if there was an obvious, significant performance gain. The third disk would be attached to the on-board SATA controller. I will discuss the recommended configuration in more detail later. Also note: the graphics card supports two monitors across any two of the three outputs, but not three concurrently. Finally, the ASUS V7-P7H55E is nearly identical in every respect. We went with the V6 based on availability.

Other laptop models
During preliminary testing we looked at the Lenovo W510, the Dell Precision 6500 and the Alienware M17x among others. All of these models were candidates that we never ruled out, but we didn’t have sufficient time with them to run the entire set of tests. However, these models had a reasonably similar configuration to the Dell Studio XPS 1645 and the Hyper-V tests we ran on these systems yielded similar results to our test model.

Other desktop models
Obviously a barebones system won’t appeal to everyone as a business solution, and it took me some time to persuade myself that it might be suitable for these environments. It wasn’t until I actually priced up this model and compared it to the comparable Dell T1500 (+~£600) and HP Z200 (slower than either model, and pricier) that I considered how it might work for us more seriously.

What am I examining, and not examining?

We have an old laptop, a new laptop, a new desktop and the cloud. Excepting the cloud (which is fixed), we’re permuting each of these hardware options with VMWare Workstation and Hyper-V test results. We’re then adding tests to examine the impact of spindle/bus speeds and the impact of adding/removing cores to these VMs. Ultimately, I wanted to quantify the productivity impacts of a change to our hardware and/or virtualisation technology as opposed to a change within our virtualisation technology, insofar as these tests could be decoupled.

I am not examining every virtualisation solution nor every hardware permutation but I do try to account for a number of these variables with these tests. I would love it if people carried out similar tests on their environments to help build knowledge in an area that’s hugely uninspected today. These are some of the other tests that I hope to revisit next year:

• The impact of application pooling on first page load times. Preliminary tests suggested there might be a small impact, but nowhere near as significant as the User Profile Service Connection. This warrants further inspection.
• The performance of “local systems” on this same hardware. As I mention above, these tests had to be de-prioritised, but I feel it would be worth identifying if there are any of these development-specific tasks where some, or all virtual technologies suffer.
• While I am running tests against a number of disk buses and configurations, I did not get the opportunity to test SSD performance. Obviously a lot of people will want to know the impact of SSD on these timings, but unfortunately I won’t have an opportunity to inspect that until early next year at the earliest.
• In some cases we work with deep snapshot trees. I want to gain an understanding of how differencing across ten or more files impacts performance for these tasks.
• Compare performance of a higher-clocked i5 to a lower-clocked i7 at a similar price range and potentially explore over-clocking options.
• Compare slower memory on an otherwise-identical system.
• Run VirtualBox tests on an otherwise-identical system.
• Assess the impact of virtualisation optimisations.

Obviously these tests say nothing about the usability of the system, power costs, mobility and more. For the purposes of this post I’m only concerned with outlining how I tested system performance for these real world tasks. In the next post, at long last, I will share the results.

For clarity, I will quickly state the problem as I see it. SharePoint 2010 system requirements and practitioner mobility requirements are inherently at odds. What guidance exists for this unique problem space tends to regurgitate preferences/allegiances rather than comparing technologies and ratifying assumptions with real-world tests. At best, you get system performance indices for a single laptop model, but these results may vary when any hardware component is changed.

How can virtualisation improve system performance?

It doesn’t. People look to virtualisation to solve other problems. However, SharePoint 2010 performs differently in different virtualisation technologies, and the margins of these differences vary by hardware configuration. By all means, the advantages of virtualisation often make it a desirable choice, but these performance characteristics need to be accounted for, lest system performance losses negate the productivity improvements that virtualisation can introduce.

Why virtualise?

There are a number of advantages to virtual systems over physical systems. Many of these benefits can also be obtained with sufficiently mature systems management technologies and physical systems, but these benefits are often easier, quicker or less costly to implement through virtualisation. Some of the benefits include:

• Provisioning times for new SharePoint environments.
• Standardisation through cloned, network-isolated virtual machines.
• Account for volatility with snapshots.
• Standard builds per-project, to share with team members, reducing project initiation costs.
• Virtual appliances produced by Microsoft and third parties, such as the Information Worker Demo VM.
• Reduced hardware rebuilds by removing development tools and SharePoint from the host.

This list is by no means comprehensive. As I say, many of these benefits can be realised with scripting and/or management tools. This list is only intended to illustrate why it’s a powerful design option.

An overview of virtualisation and related technologies

Some example technologies by type:

• Type I Hypervisors
  • VMWare ESXi
  • Hyper-V
• Type II Hypervisors
  • Oracle VirtualBox
  • VMWare Workstation
• Infrastructure as a Service (IaaS)
  • Amazon EC2
  • Azure VM Role (forthcoming)
• Local Systems
  • Native Boot Windows 7 (virtual hard disk)
  • Citrix XenDesktop (VDI)

Note: Virtual PC was not included because it doesn’t support 64-bit guest operating systems. SharePoint 2010 only runs on 64-bit systems.

Some of the alleged benefits of these approaches:

• Type I Hypervisors
  • Better performance**
  • Good management options/tools
• Type II Hypervisors
  • Host Operating System
  • Easy to use
• Infrastructure as a Service (IaaS)
  • Pay as you go
  • Scalability
• Local Systems
  • Good performance
  • Simple to use

Some of the alleged drawbacks of these approaches:

• Type I Hypervisors
  • No Host Operating System***
  • Driver issues*
  • Complicated
• Type II Hypervisors
  • Historically poor performance**
  • Historically, less manageable (snapshots, import/export, etc)
• Infrastructure as a Service (IaaS)
  • Requires stable connectivity
  • Complicated
  • Pay-As-You-Go requires diligence
• Local Systems
  • Easy to damage
  • Slow to rebuild

*Hyper-V has driver issues on some newer laptops. These are most noticeable with graphics, although I have seen audio driver problems as well. Some of these driver issues may be fixed or alleviated in the SP1 Beta/RC for Windows Server 2008 R2.

**This performance bias is one of the things I will be examining in more detail in later posts.

***This is only “sort of” true for Hyper-V, which invokes a “parent partition”. This is a special type of virtual machine that fulfils a similar role to a host operating system, and is often referred to as such.

Why are “Local Systems” included?

I’ve lumped these in for two reasons. 1) They share some characteristics with the other virtualisation technologies, like running from virtual hard drives. 2) By virtue of being local systems, they fundamentally negate some of the benefits that are obtained through virtualisation. Cloning these machines is not an option if SharePoint is installed and configured. It will be necessary to invest in scripting environment provision in order to retain those productivity benefits. It happens that many people choose to take this scripting approach, but it’s worth pointing out that network isolation and cloning can achieve similar results through virtualisation, and this does not obtain with Local Systems.

What about shared, hosted development environments?

In this scenario I’m thinking of hosted development farms, where some or all members of a team use a single environment. Based on my subjective reading of the community, this option seems to be fading away. I think there are three reasons why.

1. Cost. Running development environments on proper infrastructure is expensive. Most components have been made redundant, the storage will be expensive if it performs well, the power/cooling costs are considerably more expensive than for laptops/desktops and you will need to pay people to manage the systems. Even when these costs are split across multiple developers, it’s still expensive unless resources are overcommitted, which negates productivity gains. It also tends to be more expensive to provision new environments and this process can often be an obstacle to business agility. In a nutshell, these are protections that are unnecessary for development environments. Redundancy and resilience at this level is overkill given the associated costs. The most important assets, such as code, standard images and project-specific builds can be protected separately.
2. Hive pollution. If these farms will support multiple projects, as they often do per the previous comments about provisioning, then these systems will inherently differ from the test/stage/UAT/production systems they should resemble. Core files in the hive can be altered from project-to-project, resulting in unexpected behaviour when moving code between these environments. This can seriously complicate troubleshooting and should be avoided.
3. Mobility. These farms aren’t terribly useful to people who are travelling or who are working on-site with restricted outbound connectivity.

All of this said, there are times when project-specific requirements may make shared farms a good option. It may be sensible to take another look for:

• Integration projects.
• Developing with large amounts of data.
• Projects with heavy infrastructure requirements, such as FAST.
  • Perhaps individual development environments can consume a shared FAST Service Application?

Generally speaking, I believe these resources should be provided only in these niche cases.

How is this different from IaaS?

The main differences are costs and capital. Cloud-based infrastructure services are fundamentally just virtualised hosting on an enormous scale. This scale lowers costs to a point where it may be affordable to deploy individual machines per-developer. Although in my analyses I found that IaaS would be more expensive than desktop workstations over three years, this still may be compelling when cash flow issues preclude significant one-time investment or credit flows are restricted. IaaS should also be kept in mind when specific projects require significant provisioning or investment for a short term, for instance testing in a large farm.

While providing a single cloud-based VM per-user solves the first two issues with shared development environments, mobility is still an issue. In many places, stable mobile broadband is flaky at best. Additionally, there are key architectural differences that need to be accounted for when working in the cloud, and on a Pay-As-You-Go basis. I address all of this in my series on SharePoint 2010 Infrastructure for Amazon EC2.

Which approach is best?

This is a high-level overview of the design constraints that limited my choices, before I plunged into a concrete performance review of the remaining technologies.

Local Systems
In my view, Local Systems are only a better choice if the supporting IT systems and processes are very mature and the performance benefits are clear and significant. For most development scenarios, that has yet to be proven. I’ve postponed this virtual to physical performance comparison for now, as the other benefits of virtualisation have ruled this approach out for us, but I hope to revisit it in the new year.

IaaS
IaaS has two key planning considerations. The first is fairly obvious. Outbound RDP Connectivity needs to be open whenever the systems are needed. I encourage people to consider this in some detail and pilot with many types of users before diving in. The second consideration is Pay-As-You-Go. While cloud providers often have an always-on option, it’s usually pretty pricey. The alternative is to find a mechanism to limit compute usage to when it is truly being used, without introducing usability problems. Management tools or scripting may be able to answer these problems, but no one should enter in to this process thinking it will be easy. This is not an easy option. For a more detailed consideration of these issues, refer to my series on EC2.

Type II Hypervisors
VMWare Workstation is the most mature desktop virtualisation product on the market, although in recent years VirtualBox has been gaining share. Choosing between these technologies for my tests was never going to be easy, but I reduced it to a few factors:

• I’ve never met a VirtualBox user that would complain about using VMWare but I can’t say that proposition is reversible. There are a lot of SharePoint practitioners with a strong preference for VMWare.
• VMWare Workstation has native interoperability with other VMWare assets. While VirtualBox supports the VMDK file format, it’s not quite the same thing.
• Both products are fairly inexpensive in the grand scheme of things.
• I had stability issues with VirtualBox circa version 3.14 that left a bad taste in my mouth.

Perhaps most importantly, I felt that the performance comparison of VMWare Workstation to Hyper-V would be the most valuable decision-making information.

Type I Hypervisors
Most Type I Hypervisors would not be suitable for desktop virtualisation because they don’t have a host operating system. While it would be possible to boot a guest OS and remote in to other Virtual Machines over internal networks, this is a complicated approach and the networking requirements would be enough to put off most developers. However, as mentioned above, Hyper-V is a notable pseudo-exception to this with its parent partition.

We’ve been using the Hyper-V role in Windows Server 2008 R2 for development for a little over a year now. While we have successfully capitalised on many of the productivity benefits of virtualisation through this approach, there are a few issues that have never been entirely satisfactory:

• Despite having the host OS, using Hyper-V is still complicated for non-Systems people – particularly the networking.
  • Work-around solutions for Wireless networking are fiddly.
  • Lack of self-contained NAT requires the use of Internet Connection Sharing in order to achieve network isolation, which some users struggle with.
• Lack of Sleep/Hibernate is painful for many users.
• Graphics performance is poor – particularly with large PowerPoint/Visio files, large images and video.
• Audio can also suffer during large file operations.
• Hyper-V is not ready for laptop power schemes.

Despite these niggles, we’ve continued to use Hyper-V while waiting for the forthcoming graphics/memory improvements in Windows Server 2008 R2 SP1. I would class these usability problems as significant inconveniences that sometimes manifest themselves in lost productivity – particularly with new users learning our approach.

New Problems in SharePoint 2010

Since we properly immersed ourselves in SharePoint 2010 development, negative reports about performance started to roll in. These proved hard to validate until a few months ago when my colleagues showed me first page load times after an IISRESET in excess of one minute. This was concrete and repeatable. The problem was more severe on some systems than others, but it was clearly a problem.

The performance tests I’ve been conducting have been an effort to pick apart these results in Hyper-V. Was this new in SharePoint 2010 or did it amplify something that was minor before? Do we get the same problems on different virtualisation technologies, in the cloud or is this a symptom of virtualisation itself? In my next post I’ll discuss the environments, the tests and the testing process.

Other posts in this series

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis
• Amazon VPC and VM Import Updates

When would AWS be compelling, despite the complexity?

I’ve covered most of the design topics that I feel are relevant to SharePoint 2010 on EC2 now, so it’s time to talk about why we would use it, despite the obvious complexity that it introduces. The potential benefits included:

• Scalability. This is pretty hard to question. AWS definitely scales.
• Cash flow: The On-Demand services are Pay-As-You-Go, so this clearly helps when cash is tight.
• Infrastructure costs/support: This needs to be validated. See the AWS Premium Support Pricing page for more information about the cost of platform support.
• Performance: I will be diving much deeper in to performance over the next week or two and will be analysing EC2 alongside laptops and desktops. For now I will say that it performs well, but it isn’t the best-performing solution that we reviewed. Subjectively I would say that I don’t think most developers would consider a large instance to be slow.
• Availability anywhere (with an outbound RDP connection): Obviously the down side here is that this connection isn’t always available or reliable everywhere, for instance on a train.
• Special scenarios: Some examples I can think of here would include testing for large farms and office moves. I shan’t delve in to the scenarios, but there are sure to be others.
• Cost: This needs to be validated, and I will share an example analysis below.

Actual invoice data

This screen shot of an Amazon invoice (tidied up in Excel a bit) is the real invoice I received for my testing time. I’ve included it here because I think it illustrates the impact of instance usage time on total costs really well. It’s by far the largest cost at ~90% of the bill for this testing time, and that included a couple of weeks when I wasn’t using the instances. During that “down time” I was still billed for storage use and Elastic IP address disuse. Keep that in mind, as you will continue to accrue charges even if you shut down your machines.

Example of costs over three years

I projected charges based on these figures over three years for a large number of users. There were two main objectives for these calculations:

1. Gain an understanding of the impact of on-demand usage compared to reserved instance costs.
2. Assess these costs relative to hardware costs over an average lifetime of three years.

This analysis was only intended to indicate ballpark costs and some of the figures are nothing more than educated guesses, but I think they should serve their purpose as indications. This analysis didn’t factor in costs for Amazon’s Cloud Watch (monitoring and reporting), Amazon Support, licenses (other than Windows) and probably some other factors I overlooked, but I’m publishing it here as it might be useful for other high-level assessments. But obviously everyone should work this out for their own usage patterns and obviously, all price information is subject to change.

Summary
For seventy users, costs could break down as follows:

• 8 hours/day on-demand = $1950/instance over three years.
• 24 hours/day on-demand = $5295/instance over three years.
• Mixture of 50 reserved instances running 24 hours/day and 20 On-Demand Instances running 8 hours day = $3779/instance over three years.
• 24 hours/day reserved instances = $4511/instance over three years.

The difference in cost between on-demand usage 8 hours/day vs. 24 hours/day is enormous. Even the difference between 70 on-demand instances at an average of 8 hours/day compared to 70 reserved instances is huge: reserved instances are more than twice the cost. A mixture of reserved instances and on-demand usage probably won’t help enough to make it compelling. The only way that EC2 appears to be cost effective for large instances, used routinely, is to manage usage effectively. The detail for these calculations is provided below, with monthly costs. Those totals have been multiplied by 36 months and divided by 70 users for the cost summaries above.

An important note regarding Reserved Instances and cash flow
Reserved instances have an up-front cost of $910 per-instance per-year (or $1400 per-instance per-three-year-commitment) before usage charges are included (at a lower, reserved rate of $.24 per-instance per-hour). This means that reserved instances are a lot less viable for organisations looking to EC2 for cash flow benefits. The figures above were calculated using the $910 up-front cost, as I don’t believe most people will commit to three years of usage from the start. The reserved instance prices would clearly come down quite a bit with that three-year commitment, so feel free to recalculate as you like, but keep in mind that higher up-front cost is even worse for cash flow.

Cost assumptions

• 70 instances x 8 hours = 560 instance hours
• 560 instance hours x 230 days = 128,800 instance hours/year
• $0.48 per-hour per-instance
• Unattached Elastic IP Charges = $.01/hour unattached = 16 hours unattached/day/instance
• Elastic IP Address remap charges = first 100/month free, then $.10/remap
• EBS Storage = $.11/GB/Month x 50GB average storage
• Data Transfer In at $.10 per GB
• Data Transfer Out at $.15 per GB (first GB/month free)
• Reserved instance up-front costs of $910 for one year rather than the $1400 3 year commitment
• Bandwidth charges have not been calculated for VPC connections

Calculation Detail
On-Demand Costs at 8 hours/day average instance usage

• Instance cost = $61,824
• 16 hours unused Elastic IP charges/day = $.16 x 70 users x 230 days = $2576
• 1 remap/day x 70 users x 230 days = 16,100 remaps -1200 free = 14,900 x $.10 = $1490
• EBS costs:
  • Storage: $5.50/instance/month (assuming 50GB storage) = $840
  • IO: $5.50/instance/month (assuming roughly equivalent IO costs – perhaps 30-40 IOps) = $840
  • Snapshot Gets: costs should be negligible ~$100/year
  • Snapshot Puts: costs should be negligible ~$100/year
• Data transfer:
  • In: 20GB = $2.00/instance/month = $1680
  • Out: 20GB = $3.00/instance/month = $2520

Total = $71,970/year = $5997.50/month = £3793/month*
*This total does not include any support costs and is based on un-validated assumptions.

On-Demand Costs at 24 hours/day average instance usage

• Instance cost = $185,472
• 16 hours unused Elastic IP charges/day = $.16 x 70 users x 230 days = $2576
• 1 remap/day x 70 users x 230 days = 16,100 remaps -1200 free = 14,900 x $.10 = $1490
• EBS costs:
  • Storage: $5.50/instance/month (assuming 50GB storage) = $840
  • IO: $5.50/instance/month (assuming roughly equivalent IO costs – perhaps 30-40 IOps) = $840
  • Snapshot Gets: costs should be negligible ~$100/year
  • Snapshot Puts: costs should be negligible ~$100/year
• Data transfer:
  • In: 20GB = $2.00/instance/month = $1680
  • Out: 20GB = $3.00/instance/month = $2520

Total = $195,618/year = $16,301.50/month = £10,295/month*
*This total does not include any support costs and is based on un-validated assumptions.

Reserved Instances – Costs at 24 hours/day instance usage

• Instance cost ($910 x 70) = $63,700 + (70 x 24 hours = $92736 full-time usage) = $156,436
• 16 hours unused Elastic IP charges/day = $.16 x 70 users x 230 days = $2576
• 1 remap/day x 70 users x 230 days = 16,100 remaps -1200 free = 14,900 x $.10 = $1490
• EBS costs:
  • Storage: $5.50/instance/month (assuming 50GB storage) = $840
  • IO: $5.50/instance/month (assuming roughly equivalent IO costs – perhaps 30-40 IOps) = $840
  • Snapshot Gets: costs should be negligible ~$100/year
  • Snapshot Puts: costs should be negligible ~$100/year
• Data transfer:
  • In: 20GB = $2.00/instance/month = $1680
  • Out: 20GB = $3.00/instance/month = $2520

Total = $165,582/year = $13,882/month = £8771/month*
*This total does not include any support costs and is based on un-validated assumptions.

Mixture of On-demand and Reserved Instances – Costs at 24 hours/day instance usage
Mixture of 50 Reserved Instances and 20 On-Demand Instances.

• Reserved Instance cost ($910 x 50) = $45,500 + (50 x 24 hours = $66,240 full-time usage) = $111,740
• On-Demand Instance cost at 8 hours/day average instance usage = 20 instances x 8 hours x $.48 x 230 days = $17,664
• 16 hours unused Elastic IP charges/day = $.16 x 70 users x 230 days = $2576
• 1 remap/day x 70 users x 230 days = 16,100 remaps -1200 free = 14,900 x $.10 = $1490
• EBS costs:
  • Storage: $5.50/instance/month (assuming 50GB storage) = $840
  • IO: $5.50/instance/month (assuming roughly equivalent IO costs – perhaps 30-40 IOps) = $840
  • Snapshot Gets: costs should be negligible ~$100/year
  • Snapshot Puts: costs should be negligible ~$100/year
• Data transfer:
  • In: 20GB = $2.00/instance/month = $1680
  • Out: 20GB = $3.00/instance/month = $2520

Total = $139,550/year = $11,629.17/month = £7348/month*
*This total does not include any support costs and is based on un-validated assumptions.

Findings

Amazon Web Services can be quite expensive if usage is not controlled effectively. Based on these calculation, I don’t feel that $1950/instance over three years is bad value. These environments perform well and provisioning is very quick. There are no underlying virtualisation support costs or power costs. The scalability is inherently appealing and the Pay-As-You-Go model might be compelling for some businesses or independent users, despite all other considerations. In our case, we dove deeper in to the productivity question, attempting to get a handle on how the performance of server-class hardware in the cloud stacks up relative to laptops and desktop workstations. We wanted to understand if the complexity, potential remote worker issues and cost might be justified based on productivity gains. Some of these findings were surprising, as I will reveal in my next series of posts on SharePoint development environment performance.

Other posts in this series

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis
• Amazon VPC and VM Import Updates

Administration, Delegation and Usage Costs

The Tools

Unfortunately, the AWS Management Console user experience is fairly hideous. It doesn’t size properly in the browser and it has annoying synchronous post-back behaviours. It generally feels like an enormous Java app. I’m reminded of Cisco administration consoles circa the early part of this century. However, there is an Add-on for Firefox called ElasticFox which improves things a bit, but I wouldn’t say I’m thrilled with it either. I would classify it as less clunky, but I’d hesitate to go much further.

The AWS Management Console

ElasticFox

My colleague Brendan Newell co-evaluated Amazon Web Services with me. He identified we would need a more sophisticated management tool very early on. He found LabSlice and we looked at that for a bit. It’s fairly basic, but it adds some functionality that makes it compelling by comparison: policies, delegation and reporting. Those features provide administrative controls for smart delegation, or at least a start towards that control. It is a new product, so it’s reasonable to expect that it will improve. If we ever use AWS in anger, LabSlice or a tool like it will almost certainly form a part of the picture unless the Amazon administrative tools improve by then.

Why are these added features so important?

The underlying issue is that it’s more than twice as expensive to run an instance 24/7 than at 40 hours/week (at on-demand prices). Amazon provide Reserved Instances to try to address this always-on option, but the cost savings of “nearly 50%” assume the instance would always be running at On-Demand costs – so you’re paying for 50% of four times as many hours at On-Demand prices. This doesn’t really compute.

In reality, it may not be possible to run instances for only 40 hours/week, but it should be possible to run them for less than 50 hours/week for most users, with the right controls in place, and this figure could be a lot less if instances aren’t used every day.

So the question becomes how usage can be controlled without disrupting the value of the service. Too much control and the service becomes an obstacle to delivery. Too little control and the accounting department will be most displeased.

At a high level, these are the options we considered (with some bad options thrown in to illustrate the point):

• Get a reporting tool that will expose usage patterns on an individual and team level.
• Potentially bill teams for usage.
• Potentially bill clients for usage (trickier).
• Potentially set up a scheduled task that will automatically shut down an instance eight (or nine, or ten) hours after it is launched. Train users how to cancel the shutdown when they will be working late. While this solution is quite inelegant, it might work – depending on the users and their usage patterns.
• Use LabSlice (or a similar tool) to allow users to turn machines on and off, but not to create images or provision new machines. Set up policies to automatically shut down machines after a specific amount of running time.
• Get Draconian and have managers/administrators enforce shut down at the end of the day. Keep in mind, this is likely to taint any positives associated with this service and could prove very difficult to implement if users have valid reasons to leave machines on periodically. Is the enforcer really going to understand these nuances? In short, I suspect this won’t fit the culture of most businesses.

Remember, the point of all of this is to achieve the lowest cost, as the service will probably only be affordable with these controls in place. Without a mechanism to ensure machines are turned off, the business is exposed to 24-hour usage costs. I will give examples of projected costs without these controls later.

Back to the Public IP Addresses

Update 17 March 2011: the information regarding the public IP addresses and the VPC below is now out of date. Please see my follow-up post on Amazon VPC and VM Import Updates for more information.

If you recall from the last post, I mentioned that new public IP addresses are generated for instances whenever they are started up (unless the VPC is being used, in which case there is no public IP address). One of the features that you’ll want to find in your management tool is the ability to connect to instances after users have started them up. This environment isn’t going to work very well unless users can find out their new IP address every morning. As I mentioned before, this could also probably be scripted and is likely to form a part of other tools besides LabSlice. The point of reiterating this now is that it’s key functionality in a management tool and it will probably be very messy getting by without it.

Reporting

The last benefit of a good administration tool is reporting. If users are routinely forgetting to turn machines off, you want to know about it. If users aren’t using this system you probably want to know about it too. How are they circumventing this approach, and why?

I don’t think delegation can work without the reporting element, unless shut down policies are very effective and don’t cause disruption by terminating active sessions. Keep in mind that accountability is much less of a problem when clear, quantifiable costs can be attributed to actions. I think the ideal balance is probably high visibility of reports and delegation of start/stop functionality, potentially coupled with liberal shut down policies – perhaps at 12 hours of usage. Lastly, it should be clear that any of these approaches would need to be piloted.

Licensing

As mentioned in the first post in this series, Windows license costs are built in to instances and Amazon charges for instances based on the type of license they provide. The only license that must be paid for from Amazon is this Windows license and it is built in to the Pay-As-You-Go instance costs. If the instances are used for development then MSDN/Technet or other purchased licenses can be used in these environments for all licenses other than Windows, so long as the type of use is compliant.

Amazon offer an image with SQL built in to it. You will probably want to avoid use of this instance if you already have a SQL license, as it is considerably more expensive to run. The cost of a large Windows Server 2008 instance increases from $.48/hour to $1.08/hour accordingly. This is huge even if these numbers look small. There are 26,280 hours in three years. That’s more than $5,000 more expensive per-year (per-instance).

One thing that looked promising (until we realised it was only open to users in America) was the “Bring Your Own License” pilot. The program seems to be closed now, but I imagine this option would be interesting for readers of this blog, should that program ever form a core part of the offering, internationally. This of course assumes that subtracting Windows license costs from the instance charges results in a significant saving.

Recommendations

The main contentious issues for which there is no clear, one-size fits all guidance are topology, network configuration and management. We were looking at an all-in-one server, including the DC/DNS roles, on private and public dynamic IP addresses with considerable piloting in this configuration, supported by LabSlice. The costs of the management tool are going to be insignificant relative to what it saves you, even if you write it yourself. I consider it to be fairly indispensable, with the possible exception of Reserved Instances at the three-years up-front cost of $1400/instance (plus usage). I will explore these cost specifics in greater detail in my next post.

Other posts in this series

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis
• Amazon VPC and VM Import Updates

Machine names, Domain SIDs and Cloning

In our testing, we were able to run multiple instances of the same AMI concurrently, which can be desirable if you have a team of developers with similar or identical requirements. We could run these instances beside each other without conflicts because we had all roles (including the DC/DNS) on one machine and we locked down the firewall, which is advisable anyway in the cloud. We only allowed the RDP port inbound to start with, and opened HTTP/HTTPS traffic where it was helpful to do so. This cloning story would get much more complicated with multiple servers, as I discuss in more detail in the networking section below.

One big “gotcha” in this area is the default settings of the EC2 Service Properties when Amazon’s Windows AMI is launched initially. This is one of the few additions that Amazon packages with their Windows image. In the EC2 Service Properties you should de-select the Set Computer Name and Set Password options. The Set Computer Name option seriously causes problems for SharePoint, as it changes the Machine Name whenever the instance is started up. The good news is that you only need to do this once if you will be creating a new base image. Just be careful not to change this setting back later on.

The EC2Config Service

Networking

By default, Amazon assigns a public IPv4 address to EC2 instances via DHCP. This IP address changes whenever an instance is launched, allowing Amazon to manage their pool of public IPv4 addresses effectively. Until IPv6 adoption ramps up, this is the only viable option for an offering of this scale, although Amazon are actively looking at IPv6 today. By default, Amazon also assigns a private IPv4 address to EC2 instances via DHCP. This internal IP address also changes whenever an instance is launched.

Internal and external dynamic IP addressing introduces considerable design complexity for SharePoint development environments. This complexity is heightened by the addition of Elastic IP Addresses and the Virtual Private Cloud options.

Domain Controllers and Private DHCP

As I’ve mentioned before, SharePoint 2010 development environments need to be members of a domain in order to successfully deploy the Search or User Profile Service Applications, but unfortunately dynamically-assigned IP addresses and domain controllers don’t play nicely together. I shan’t delve in to those details much here, but this has been known to cause problems with start-up times for DCs, and member servers won’t know how to find the DCs once the DC’s IP address changes. Additionally, there are Firewall policy implications.

With the exception of the Virtual Private Cloud (discussed below), we had to rule out persistent multiple-server farms for these reasons. The complexities of managing this stuff on a daily basis would be beyond most users and would probably create system instability or at the very least, add cost (by leaving the DC on all the time). The option of adding a second DC for resilience and to possibly work around some of these issues would add further complexity and cost. Basically, this wasn’t working.

Developing on Domain Controllers

The only viable approach we could find for working with DCs on DHCP was to make the SharePoint development machine the domain controller. This is a step backwards in many ways, as this configuration has been known to cause issues. As I summarised in March 2010 (from the link above):

• Domain Controller security is bad for development. It means developers will be coding as Domain Admins and they will be doing so on a machine with Domain Controller security policies. This is just a mess. It’s tighter than it should be in some respects and looser in others.
• SQL doesn’t like to run on a DC.
• Running a DC, SQL and SharePoint on the same machine creates a massive load of service start-up contention and sometimes the environment will start from an unstable point because dependent services will not be ready when a depending service tries to start.
  • This also increases start-up time considerably.
• Adding Visual Studio to this mix causes known performance issues. The machine simply can’t keep up with doing all of this.

Having said all of that a while ago, based on lesser-performing equipment, we didn’t actually find that performance or installation were particularly troublesome on EC2, although we did encounter a security policy issue or two. I still have reservations about the code quality that will emerge from development on a domain controller, but if this is acceptable for your requirements then I think this is the most significant Private DHCP issue conquered. If not, you will probably need to look at the Virtual Private Cloud. Other topologies are conceivable but with even more complexity than we’re already contending with. These are unlikely to be broadly usable.

Public DHCP issues

The primary issue with dynamic public IP addresses is finding out what the new address is. This is easy enough if you have access to the AWS console, as you can pull the new address directly from the instance description and even download a file to launch an RDP connection to the new IP address directly. However, it’s very unlikely that it will be acceptable to give access to the AWS Management Console to all users. This leaves three options, as I see it:

• Leave the machines running 24/7 (at a potentially massive increase in cost).
• Have an administrator send/provide the addresses to users as the instances are started up.
  • This feels very clunky to me and untenable in the long term.
• Find a management tool (there are a few) or a scripted approach to handling this scenario.

Whatever the solution, it’s likely to form part of the broader question of administration, management tools and delegation, which I’ll come back to in the next post. I believe this can be solved without too much difficulty, but it requires some thought along these lines in order to avoid a mess.

Elastic IP Addresses

One way that Amazon has tried to ease the pain of Public DHCP is the Elastic IP Address. By default, each customer is given five, although you can request more. Elastic IP addresses are applied to an instance while it’s running. A few minutes after it has been applied it takes over from the DHCP-assigned address and users can access the instance at their usual address. However, this requires intervention by an administrator to associate the Elastic IP Address with the instance after it’s been started. Alternately it can be scripted. Just keep in mind this is another option that probably isn’t best delegated to everyone by giving all users access to the AWS console.

One thing that’s particularly crafty about Elastic IP Addresses is that you are charged $.01 for each hour they are not in use. If you’re diligent about turning off your machines when you’re not using them, you will get nailed for not using the IP address. Granted, it’s a small charge and with IPv4 address supplies dwindling very quickly, perhaps not that unreasonable.

In my view, Elastic IP Addresses probably aren’t going to solve a lot of problems, but in some cases it may make things easier – particularly if pointing DNS at these addresses.

Virtual Private Cloud

Update 17 March 2011: the information regarding the public IP addresses and the VPC below is now out of date. Please see my follow-up post on Amazon VPC and VM Import Updates for more information.

The Virtual Private Cloud (VPC) is effectively a VPN connection between your network and AWS. It allows fixed private IP addresses, DHCP options like DNS/WINS servers, and allows you to connect existing assets to the cloud, for instance management or backup servers. This may also help with SSO. I shan’t belabour the design options for the VPC, because at face value it should be pretty obvious if it’s the right fit for your uses. There are obvious security considerations about opening up this communication across the WAN to a third-party, but that’s not to say there aren’t ways it can be set up well – for instance creating a dedicated domain in the VPC.

The most important thing to know about the VPC is that when instances are launched they only get a NIC with an IP address on a VPC subnet. There is no public IP address for the instance. This means the only way you can access the instance is via the other end of the VPN (typically the corporate network). This may introduce some funky routing and potentially degrade speed/reliability for users working from home or on client sites. On the other hand, it may not. It’s critical that this option is thought through with a broad design team including internal network and systems teams. I would highly recommend testing/piloting this configuration as well (noting that the initial configuration may be expensive for a test, since it will integrate with production infrastructure).

I think the VPC can answer a lot of the shortcomings of the standard EC2 IP addressing approach if public IP addressing is not a requirement. I’m not sure why NAT couldn’t have been used to allocate fixed internal addresses by default, but it hasn’t been, so we’ve only got one way in to the VPC. Once in it, you can deploy single-server machines as we did without the VPC (assuming the firewall is locked down in the same way), or join SysPrep’d SharePoint servers to a shared domain infrastructure. This assumes SharePoint provisioning (scripting installation/configuration) is mature enough that manual configuration steps don’t impede productivity. Other topologies may be valid as well. In principal it shouldn’t be miles different from your LAN. The main things to understand is that the private IP addresses are assigned by Amazon and there is just the one way in. Note: there’s quite a bit to understand about planning the VPC itself, and pricing for that traffic, which is all outside of the scope of what I’m inspecting here, so please refer to the Amazon VPC resources for more information. Also be aware that it’s still in Beta.

Networking can be enough to melt anyone’s brain, so I’ll save administration, delegation and licensing until my next post.

Other posts in this series:

• SharePoint 2010 Infrastructure for Amazon EC2 Part I: Storage and Provisioning
• SharePoint 2010 Infrastructure for Amazon EC2 Part II: Cloning and Networking
• SharePoint 2010 Infrastructure for Amazon EC2 Part III: Administration, Delegation and Licensing
• SharePoint 2010 Infrastructure for Amazon EC2 Part IV: Cost Analysis
• Amazon VPC and VM Import Updates

What are the Amazon Web Services?

AWS is a platform in the cloud, like Windows Azure in some respects. While these web services are distinct from traditional hosting offerings, Amazon also provides Infrastructure as a Service (IaaS) in the form of Elastic Cloud Compute (EC2). This is a Red Hat implementation of the Xen hypervisor, from which virtual machines (instances) can be launched. For accuracy, I should note that Amazon recently launched a second Oracle hypervisor within EC2, but that’s a distraction from this discussion. Amazon have been providing their web services since 2006. For the purposes of these posts I am concerned with the EC2 offering as a cloud-based alternative to desktop development workstations, although there are other scenarios that may be suitable for deployment in EC2, such as demonstrations or large infrastructure tests. For more information on the difference between traditional hosting and EC2, see Amazon’s FAQ on the matter.

What is Elasticity?

This term arises frequently in the Amazon vernacular. In its essence this means that scalability is built in to the platform. Need more CPU or memory? Just re-launch your instance as a larger size. Need more instances? Create them in a few minutes. Need more storage? They got it and then some. IP addresses? They even have Elastic IP addresses. Bandwidth? It’s the cloud, fool.

AWS largely deliver on these promises, although you’ll encounter some provisioning fiddlery before realising it. More importantly, increased size comes at a cost. Nearly all of the Amazon price points are ridiculously low at their smallest, but these costs are not always linear – particularly with CPU and memory. Additionally, cost permeates nearly every design option, and these costs persist over time. Infinitesimally small prices need to be considered over very long periods if IaaS is to become an alternative to hardware. I will discuss costs in more detail later, as this topic is fundamental to the desirability of cloud computing. If it isn’t cost effective, it probably won’t be the right option. But in a nut shell, Elasticity means that if you need more of anything, you can pay for it for the duration that you need it. They definitely intend to say that you can shrink as well.

EC2 Design Complexity

I couldn’t possibly hope to explain everything that’s important to know about AWS in these blog posts and I won’t try. However, it’s important to know that the design constraints that pricing and scalability impose on AWS require a fresh perspective.  Infrastructure Architecture for AWS will require time, testing, piloting and a good understanding of end-user working patterns. Once this configuration and these patterns are clearly understood, the costs need to be projected over long periods. This is likely to be a deep consulting exercise, since so few design options can be left to chance; this will hopefully become clearer as I talk more about pricing later. For now, if you don’t believe this is complicated, have a look at the 237 page User Guide, which I would class as required reading for anyone serious about EC2. The topics covered below are a summary of the areas that I feel are most important to understand with SharePoint on EC2.

Storage

The first thing to understand about AWS is that there are two types of storage, the Simple Storage Service (S3) and the Elastic Block Store (EBS). Some older documents and forum posts were written before EBS was available as a root device, so watch out for potentially misleading information.

All SharePoint 2010 environments need to run on EBS because Windows Server 2008 will chew up more than 10 GiB off the bat (this is the maximum size of S3 volumes). EBS storage costs are more expensive than S3 and you pay for the number of I/O requests, so projecting costs is a fairly inexact science. However, in my brief testing time the I/O charges were relatively small. It’s worth noting that for the extra cost, EBS volumes also persist and they launch faster. I am only briefly touching on this topic, so please review the User Guide if this is insufficient detail. The key points for now are that you must use EBS for SharePoint instances, and EBS is more expensive than S3.

Provisioning
Taking snapshots and creating new images from them is quick and easy in EC2, once you get your head around the key concepts: AMIs, Volumes and Snapshots.

AMIs
An AMI is an Amazon Machine Image. This will be the first design choice you encounter when launching an Instance. Amazon provides a basic Windows Image or you can use an Amazon image with SQL included (at a cost). You can use your own licenses for everything but Windows.

Once an image is running you can modify it to your taste. Once you’ve created a new standard baseline, you can create a new image from your instance, and when you provision new instances you will be able to select this new image rather than the Amazon one you started with. Note: the Amazon Windows license cost is built in to the billing process; your instance costs include the license, even after you’ve created your own new image from the original. Also note: Windows Server 2008 R2 is not available yet.

Volumes
A volume is basically a virtual hard disk. When a new instance is created, the selected AMI is deployed to a new volume – the same size as the image it was created from. A volume can only be attached to one instance at a time, but an instance can have many volumes attached to it if you want to add storage capacity.

Remember that you pay for the storage you use, so size your volumes wisely. 30 GiB is unlikely to last anyone very long with Windows Server 2008, so consider at least 40, if not 50 GiB for any new root volumes. Keep in mind, you may find the less expensive S3 volumes useful as secondary, disposable storage if that suits temporary needs.

Snapshots
A snapshot records the state of a volume at a point in time. Once a snapshot has been taken, a new volume (of equal or greater size) can be created from the snapshot and that new volume can be attached to a new instance. That new instance can be used to create a new AMI at the new size. Snapshots and new volumes together enable you to increase system disk size. Snapshots can also be used for backup.

An example workflow for getting your first image at the right size might go like this:

• Launch an instance from the default Windows Server 2008 image.
• Install SQL and SharePoint (this should be possible at just under 30GiB).
• Configure stuff and shut down the instance.
• Take a snapshot.
• Create a new volume at 50GiB based on the snapshot.
• Detach the existing volume from the instance and attach the new volume.
• Create an image from the instance.
• Launch the existing instance and create additional instances from the new AMI as needed.

Note: if you will be including Visual Studio or any other sizeable software, you will need to go through a process like this before installing it, as it will push you over the 30GiB initial size.

This process is oversimplified, but it hopefully illustrates the relationship between AMIs, snapshots and volumes as they relate to provisioning. All told, I think this way of working with images, volumes and snapshots is sensible, not terribly complicated in the EC2 scheme of things, and the choices should be pretty straight-forward once you understand the options and costs. However, this could potentially get more complicated as end-users engage with these decisions. How will they know what to ask for? Will it be necessary to involve EC2 experts in the approval of any new systems? Training, consultancy or winging it all have associated costs and risks. Even though I’m only talking about development environments here, there are still risks  in committing to a Pay-As-You-Go platform where usage is unrestricted. Keep this in mind.

I’m aware this is lengthy already, so I’m going to split this up. In my next post I’ll review Cloning and  Networking.

First Page Load After IIS Reset in SharePoint 2010

One of the key performance indicators I’m measuring is first page load after an IIS reset.

Why not just do an Application Pool Recycle?
Before going any further, I acknowledge that most developers will be able to save a lot of time by recycling application pools rather than resetting IIS – but there are still scenarios when a full IIS reset is required and we’re finding that first page load after an IIS reset is a great deal slower in SharePoint 2010 than it was in 2007. First page load has always been notably slow and people have written warm-up scripts to address this scenario post-reboot, but in SharePoint 2010 I’m noticing speeds are two or three times slower.

I’d initially hoped that I could use this long first page load time to my advantage, which partially explains the time I’ve spent working on this issue here. I was thinking, surely if it takes so long, that extra time will give me a more accurate measure of these performance indicators across different systems. However, as I started to test on server class hardware I was finding that the performance gains were by no means linear and much less than I would have expected. This also held true with i7 laptops, i7 desktops and Amazon EC2. Interestingly, it appeared that the CPU was in no way fully utilised on any of these systems when loading the page for the first time, and these timings did not improve by adding additional CPUs. Earlier tests suggested that disk speed was not a significant factor in first page load times and memory is in no way constrained during these tests.

The Speedy (but evil) White Wizard

In the second instance, we noticed that not all farms were as slow as most of them seemed to be. We stumbled across this accidentally when testing performance in Amazon Web Services (AWS). A colleague did our initial AWS work and we were both very impressed by the initial performance results. A few days later I joined in the fun and built my first single-server instance. We immediately noticed that my first page load times were approximately double the times that my colleague was seeing. Eventually we identified that he used the Farm Configuration Wizard while I had manually created a separate Application Pool for each of my Service Applications. This warranted further investigation.

A note about application pooling
My approach to creating a separate Application Pool for each Service Application is to some extent a hang-over from SharePoint 2007 least-privileged thinking. I was aware that this approach exceeded recommended Application Pool capacity limits, but I didn’t let this trouble me too much based on the single-user load; I’ve always prioritised adherence to the least-privileged model over minor performance degradation. However, based on these seemingly significant performance results, emerging community consensus and the best guidance available today, I decided to reconsider this approach.

A note about AWS
There are a number of broader architectural challenges to conquer when designing a SharePoint 2010 development environment in AWS, which is a topic that I hope to return to in a later post.

A note about the Farm Configuration Wizard
This is the page that greets you immediately after installing SharePoint. It takes care of a lot of Services and Service Applications in one go, but it does some pretty undesirable things as well. In short, for all but the most playful of applications, it’s not appropriate. Build the Service Applications properly.

Reconsidering Application Pooling

As mentioned above, my next step was to quantify the improvements that can be gained through pooling applications. My first test was to delete all of my Service Applications and re-create them in a single application pool. I also deleted all of the web applications and created them in a single, separate Application Pool.

Following my normal development environment build process, I created all of the Service Applications and the Web Applications before tackling the User Profile Service Application. Out of curiosity, I quickly tested first page load times and was happily surprised to find that they had been cut in half. So I took a snapshot and created the User Profile Service Application.

I Blame the User Profile Service Application

After creating the new User Profile Service Application and running an IISRESET, my first page load of Central Administration was almost exactly as slow as it had been with all the Service Applications in their own pools. This was before creating a synchronisation or doing anything with the newly created Service Application. It was merely provisioned. At my wits end, I called it a night.

Having thought about it some the next morning, I decided to create a new web application with a Blank root Site Collection. I already had a similar web application in my farm but I made one key configuration change to the new one. When creating the web application I created a custom Application Proxy Group and removed the Service Connection to the User Profile Service Application. I then tested first page load times on my two blank sites. The new site without the Service Connection to the User Profile Service Application loaded as quickly as the sites did before I created the User Profile Service Application. The original site loaded in the same time as the old sites. The disconnected site was approximately twice as fast to load.

Validating the Results

After reaching this provisional finding, I fired up the Microsoft Information Worker Demo VM. I wanted to test this on a completely different virtual machine (but on the same hardware). I created two new web applications with two new Blank root site collections. I ommitted the Service Connection to the User Profile Service Application on the second web application again. My timings were nearly identical to the timings on the first machines.

Next, I reverted to the earlier snapshot of my development environment – the one with each Service Application in a different Application Pool. I created a new web application with the Blank root Site Collection again and got nearly the same results. In this case, all of the results were slightly slower (a couple of seconds) than they were in my snapshot with all the Service Applications and Web Applications pooled together, but the Service Connection to the User Profile Service Application was a much bigger factor (~20 seconds).

What about the Farm Configuration Wizard Results?
You may be wondering why the sites on the Wizard-configured farm loaded quickly. While I’ve not spent any time revisiting that environment and I’ve never spent much time on servers configured by that wizard, I strongly suspect  this is because the User Profile Synchronisation Service had never been successfully provisioned.

I am still in the process of further validating these results across various hardware configurations and within various virtualisation technologies. My tests should provide better data on the benefits of pooling the Service Applications as well. All of these findings are somewhat provisional, but I’d say the Service Connection results so far are the clearest findings I’ve got to date, by a considerable margin. In short, I think you can expect first page load times to be at least twice as quick when the Web Application is disconnected from the User Profile Service Application.

But I Kind of Need That Service Connection

Touché! You often will. In fact, let me back-track and say that I haven’t really considered how these findings can be applied in the real world yet. In previous development environment iterations, we found that we needed to abandon development in a Workgroup so we could connect to the User Profile Service Application. It may be that some web applications can live without this connection (for instance, many WCM apps), but as I say, my head is deep in performance considerations at the moment and I really haven’t had time to consider these implications yet. However, I will try to revisit the topic reasonably soon and I welcome comments! One way or the other, it’s good to have a better understanding of why 2010 first page load times are so much slower than 2007.

Does this slow anything else down?

At this point, I haven’t had a chance to test much else, but I have tested creating a new web application with and without this Service Connection. No impact. I also tried creating a Publishing Portal within those web applications, and again, no impact.

If you’re curious about the actual performance figures, I hope to publish them in the next couple of weeks. To give a high-level indication, in one environment the connected First Page Load times were ~35 seconds and the disconnected times were ~17 seconds. In slower environments this difference may be even greater.

I still have yet to find the time to test this myself, but I’d be very keen to hear about your experiences – good or bad. Failing that, I hope to get back to this in the next couple of weeks.