• I have an intranet and a MySite, each with a default zone used by all users to access the application, in this example https://sp and https://my.
• I want to optimise my crawl performance by crawling over HTTP, in order to remove the encryption/decryption load for each request, so I create a new Alternate Access Mapping zone on HTTP, for each of these web applications. These new URLs are http://s and http://m.
• I also set up HOSTS file entries on my Crawl Component servers so they will be able to crawl these applications locally. I don’t want my crawls to add load to the WFE servers.
• If I don’t want real people to access the site on this zone (and I probably don’t), I don’t set up DNS entries for it. Only Search will be using this zone.
• (Optionally) I create a Web Application User Policy that restricts access to this zone, since it is not running under SSL.
• I reconfigure my Content Sources in the Search Service Application to crawl the new zones, http://s, http://m and sps3://s
  • Note, for an SSL-secured site, the final People Search Connector, “sps3://s” will actually be “sps3s://s” by default, so make sure to get rid of that last “s”.
• I run a full crawl and verify that it completes much faster than it did when I was crawling the site over HTTPS/SPS3S.
• Once my full crawl completes successfully, I verify that All Sites and People search results are returning as https://sp and https://my (the default zones).

This was all working as expected – until Anthony noticed that the Org Browser web part appeared to be broken. On further inspection, we identified that:

• Alternate Access Mappings didn’t appear to be working on this link (it displayed the crawled address).
• All other links on the People Search Results tab pointed at the default zone; they accurately respected Alternate Access Mappings.
• All results for the All Sites tab/scope successfully respected Alternate Access Mappings.

So we have one link to the wrong zone on the Enterprise Search Centre’s out-of-the-box People Search results page (or tab, if you prefer). I took this scenario to my development environment to confirm. Indeed, the behaviour was the same. I’ll illustrate below.

Alternate Access Mappings for my Blank Site

Alternate Access Mappings for my My Site

The updated Content Sources page

The “All Sites” Search Results Page With All Links Mapped

In the following four screen shots of the same People Search results, notice the status bar as I hover over various links on this page (see: bottom left of the browser).

The Correct Link to Me

The Correct Link to “Add as Colleague”

The Correct Refinement Link to “All Matches”

The Incorrect Organisation Chart Link (http://m)

It would be incorrect to say that this link is broken. It works, so long as you have name resolution for it. The link is just pointing at the wrong zone, which may not be in DNS. Further, clicking that link may not be desirable or could disrupt the user experience, if the zone is locked down. In short, we explicitly want to avoid edits from that zone in this case, for a number of reasons, all of which I will gloss over here as my preference for a single zone.

Long story short: this looks like a bug to me. I’ll call it a bug. These links are new to User Profiles in SharePoint 2010, so there’s a likelihood that Alternate Access Mappings were overlooked here.

What to do?

I still want to use Alternate Access Mappings, because the performance overhead of crawling encrypted data is not unsubstantial. This leaves me with an option to escalate this issue through Microsoft Support, which I really don’t have the time to do, or we could probably whip up some script quickly to update the results page and fix these two links up, but that’s not a very elegant solution. This is no longer a pressing need in my case, because this client has opted to delay launch of MySites and People Search for reasons completely off this map, but the bug remains and pertains elsewhere (like in my development environment). I will take this to Microsoft when I can find the time, but that’s not going to happen in the near future. Until then, any other ideas?

17-03-2011 update: David noted in the comments that he’s worked around this by running his sps3:// crawl on a different web application – even creating one with no real content, specifically for this purpose. I haven’t had a chance to test it out yet but it sounds like a great idea to me. See the comment for more detail. Anthony has put this to the test for our client and all is working now. Hooray!

Afterword: Server Name Mappings

Somewhere along the line I’ve got myself in a muddle regarding Server Name Mappings. When we first encountered this problem, I configured both Alternate Access Mappings and Server Name Mappings (with the same mappings). In this case, I don’t believe this has caused any problems, but it’s not necessary and isn’t correct. Alternate Access Mappings should translate search results to the same zone that you’re browsing from, without doing any extra work. Server Name Mappings translate crawled data like file shares to other links that don’t already exist as Alternate Access Mappings. As this Enterprise Search blog post explains, “Although Server Name Mapping and Alternate Access Mapping achieve seemingly similar results, they work independently, addressing different problems, and should not be used together”. This is perhaps the only content I’ve found that clearly explains how to use Server Name Mappings correctly, and is well worth a read.

It’s worth noting that the Server Name Mappings had no impact on the Organisation Browser link either.

First Page Load After IIS Reset in SharePoint 2010

One of the key performance indicators I’m measuring is first page load after an IIS reset.

Why not just do an Application Pool Recycle?
Before going any further, I acknowledge that most developers will be able to save a lot of time by recycling application pools rather than resetting IIS – but there are still scenarios when a full IIS reset is required and we’re finding that first page load after an IIS reset is a great deal slower in SharePoint 2010 than it was in 2007. First page load has always been notably slow and people have written warm-up scripts to address this scenario post-reboot, but in SharePoint 2010 I’m noticing speeds are two or three times slower.

I’d initially hoped that I could use this long first page load time to my advantage, which partially explains the time I’ve spent working on this issue here. I was thinking, surely if it takes so long, that extra time will give me a more accurate measure of these performance indicators across different systems. However, as I started to test on server class hardware I was finding that the performance gains were by no means linear and much less than I would have expected. This also held true with i7 laptops, i7 desktops and Amazon EC2. Interestingly, it appeared that the CPU was in no way fully utilised on any of these systems when loading the page for the first time, and these timings did not improve by adding additional CPUs. Earlier tests suggested that disk speed was not a significant factor in first page load times and memory is in no way constrained during these tests.

The Speedy (but evil) White Wizard

In the second instance, we noticed that not all farms were as slow as most of them seemed to be. We stumbled across this accidentally when testing performance in Amazon Web Services (AWS). A colleague did our initial AWS work and we were both very impressed by the initial performance results. A few days later I joined in the fun and built my first single-server instance. We immediately noticed that my first page load times were approximately double the times that my colleague was seeing. Eventually we identified that he used the Farm Configuration Wizard while I had manually created a separate Application Pool for each of my Service Applications. This warranted further investigation.

A note about application pooling
My approach to creating a separate Application Pool for each Service Application is to some extent a hang-over from SharePoint 2007 least-privileged thinking. I was aware that this approach exceeded recommended Application Pool capacity limits, but I didn’t let this trouble me too much based on the single-user load; I’ve always prioritised adherence to the least-privileged model over minor performance degradation. However, based on these seemingly significant performance results, emerging community consensus and the best guidance available today, I decided to reconsider this approach.

A note about AWS
There are a number of broader architectural challenges to conquer when designing a SharePoint 2010 development environment in AWS, which is a topic that I hope to return to in a later post.

A note about the Farm Configuration Wizard
This is the page that greets you immediately after installing SharePoint. It takes care of a lot of Services and Service Applications in one go, but it does some pretty undesirable things as well. In short, for all but the most playful of applications, it’s not appropriate. Build the Service Applications properly.

Reconsidering Application Pooling

As mentioned above, my next step was to quantify the improvements that can be gained through pooling applications. My first test was to delete all of my Service Applications and re-create them in a single application pool. I also deleted all of the web applications and created them in a single, separate Application Pool.

Following my normal development environment build process, I created all of the Service Applications and the Web Applications before tackling the User Profile Service Application. Out of curiosity, I quickly tested first page load times and was happily surprised to find that they had been cut in half. So I took a snapshot and created the User Profile Service Application.

I Blame the User Profile Service Application

After creating the new User Profile Service Application and running an IISRESET, my first page load of Central Administration was almost exactly as slow as it had been with all the Service Applications in their own pools. This was before creating a synchronisation or doing anything with the newly created Service Application. It was merely provisioned. At my wits end, I called it a night.

Having thought about it some the next morning, I decided to create a new web application with a Blank root Site Collection. I already had a similar web application in my farm but I made one key configuration change to the new one. When creating the web application I created a custom Application Proxy Group and removed the Service Connection to the User Profile Service Application. I then tested first page load times on my two blank sites. The new site without the Service Connection to the User Profile Service Application loaded as quickly as the sites did before I created the User Profile Service Application. The original site loaded in the same time as the old sites. The disconnected site was approximately twice as fast to load.

Validating the Results

After reaching this provisional finding, I fired up the Microsoft Information Worker Demo VM. I wanted to test this on a completely different virtual machine (but on the same hardware). I created two new web applications with two new Blank root site collections. I ommitted the Service Connection to the User Profile Service Application on the second web application again. My timings were nearly identical to the timings on the first machines.

Next, I reverted to the earlier snapshot of my development environment – the one with each Service Application in a different Application Pool. I created a new web application with the Blank root Site Collection again and got nearly the same results. In this case, all of the results were slightly slower (a couple of seconds) than they were in my snapshot with all the Service Applications and Web Applications pooled together, but the Service Connection to the User Profile Service Application was a much bigger factor (~20 seconds).

What about the Farm Configuration Wizard Results?
You may be wondering why the sites on the Wizard-configured farm loaded quickly. While I’ve not spent any time revisiting that environment and I’ve never spent much time on servers configured by that wizard, I strongly suspect  this is because the User Profile Synchronisation Service had never been successfully provisioned.

I am still in the process of further validating these results across various hardware configurations and within various virtualisation technologies. My tests should provide better data on the benefits of pooling the Service Applications as well. All of these findings are somewhat provisional, but I’d say the Service Connection results so far are the clearest findings I’ve got to date, by a considerable margin. In short, I think you can expect first page load times to be at least twice as quick when the Web Application is disconnected from the User Profile Service Application.

But I Kind of Need That Service Connection

Touché! You often will. In fact, let me back-track and say that I haven’t really considered how these findings can be applied in the real world yet. In previous development environment iterations, we found that we needed to abandon development in a Workgroup so we could connect to the User Profile Service Application. It may be that some web applications can live without this connection (for instance, many WCM apps), but as I say, my head is deep in performance considerations at the moment and I really haven’t had time to consider these implications yet. However, I will try to revisit the topic reasonably soon and I welcome comments! One way or the other, it’s good to have a better understanding of why 2010 first page load times are so much slower than 2007.

Does this slow anything else down?

At this point, I haven’t had a chance to test much else, but I have tested creating a new web application with and without this Service Connection. No impact. I also tried creating a Publishing Portal within those web applications, and again, no impact.

If you’re curious about the actual performance figures, I hope to publish them in the next couple of weeks. To give a high-level indication, in one environment the connected First Page Load times were ~35 seconds and the disconnected times were ~17 seconds. In slower environments this difference may be even greater.