First, consider an organisation with offices across multiple floors or buildings. Lync may be a very effective means of connecting these employees despite their relatively close proximity. If this traffic can route locally it can be a big plus – especially if there’s lots of media traffic. Second, consider an organisation with multiple branches. They invested in private WAN links to connect these branches and don’t necessarily want to route Lync traffic over their internet connections if they can avoid it. For some organisations these will be non-issues, since Lync traffic is optimised for the WAN, but for other organisations this may be important – particularly if they’re in a part of the world where internet connections are slow or expensive (or both). So we went about testing this with the Lync 2010 client and Office 365 users (the behaviour is the same with Microsoft Online IDs or federated users).

After monitoring this network traffic for a bit, I picked up on behaviour I didn’t expect to find. In this case I was making voice or video calls (sometimes referred to as “media” traffic):

• If all ports are open, media (voice/video) traffic will route peer-to-peer between the two client machines (with some important caveats to follow in a later post).
• If inbound traffic is blocked to one of the client machines (for instance by Windows Firewall or another network device), media traffic will route via the Office 365 Media Edge servers, while media traffic to the client machine that allows the inbound traffic will be transmitted directly from peer-to-peer.

I didn’t expect this would be the case, but it’s great that it behaves this way. The Lync client is smart enough to send voice/video data over two totally different routes if peer-to-peer communications are open in one direction and closed in the other. I’ll talk more about some caveats to this finding in a later post, but for now I want to consider what this tells us about this tool. Some of my assumptions about Lync were wrong and I think it’s worth pondering where those assumptions came from.

In search of a new conceptual model

At a glance, the telephone appears to be the best conceptual model for Lync, whether you’re texting, phoning or making a video call. There are even telephone icons in the Lync client, like this one:

In this telephony model, we assume there is one channel open in both directions, connected by patch panel on a switchboard. Some of us may even have an awareness of the “last mile” of cabling from a central office to personal or business premises.

Despite the obviousness of this comparison so far, we need to think about the other things Lync can do. The importance of this other functionality can be easy to overlook because this notion of a single open channel is such a deep association with voice communications – right back to the tin can and string. Even messenger communications evoke this sense of a single private channel, although we know it’s different than a phone. Perhaps this sticks in the same way because messenger often supplants a phone call and it’s nearly as immediate? Whatever it may be, the “why” is not important, but the phenomenon is.

Beyond telephony

Some of the other features we’re talking about here come from Live Meeting and other meeting workspace tools based on desktop sharing. Altogether, we’re looking at messaging, voice, video, desktop sharing, file transfers, whiteboards, presentations and polls. This looks a lot less like a phone call now. It looks more like a conference or a presentation, without having to leave your desk.

In a physical conference session, speakers may tag-team, or participants may ask questions, but typically a single participant will be sending information to the others when they have the focus. This is the essence of the medium: a group of participants observe one presenter at a time. There is one mixing desk, one source of video to the projectors, etc. If you’re participating in a conference with multiple speakers with microphones, communications suffer until a single participant regains focus. The best illustration of this point is a news panel discussion or a political debate. You get gibberish until things settle down again. Really this isn’t much different than any other form of communication, but there’s some pretty clear social norms involved.

This behaviour is clear to see in the Lync client when you have a multiple-participant video call, as the video presentation is automatically switched to the current speaker/mover. To this end, everything comes via a single source. If you prefer the technical jargon, “communication between the local and remote participants is mediated by a modality-specific multipoint control unit (MCU)“, in the Lync Server topology – in this case at Office 365. Think of this like a director in live television, who selects the camera to broadcast from many simultaneous inputs. In this case, that’s got a clever algorithm behind it rather than a person.

Now if you take this conference model and shrink it back down to two participants, while using the same tool, not a lot needs to change, but there is a new dimension to the communication – it’s private again now. Perceptually we are subject to this deeply-engrained association of a single private channel. At a user experience level, nothing really changes except that we always see the other participant because there’s no one else to switch to. At a technical level, and returning to the main point here, the communications can route peer-to-peer because there’s no need for mediation, assuming all of the necessary network ports are open. Put another way, the traffic routes in a clever way, if possible. The funky thing is that those routes needn’t be the same.

Two Strings and Four Cans

So how should we characterise this “conference”, in light of the fact that it’s private again now and that each client finds its own optimal route? To go all the way back to the tin cans and string, we need to clarify one thing. With a tin can and string, the can is both microphone and loudspeaker, but not at the same time. Now we have two strings and four cans, but each participant has one can acting as a microphone and another as a loudspeaker – each with its own piece of string. In our original example where one participant’s Windows Firewall was blocking inbound traffic, one of those lines of string has to go out the window and through the cat flap, while the other goes directly through an open door. To labour this strained metaphor further, that means you need a longer piece of string and that the messages on it will take longer to arrive.

Ultimately, I’m not sure how helpful this is as a replacement conceptual model, so I’ve contented myself with an acknowledgement that I’m synthesizing the illusion of a single channel, while smart stuff happens in the background to make sure that I get the best possible private communication over two.

There’s a few points that fall out of this at the network level:

1. It might be desirable to open up ports internally to allow this traffic to flow locally or across private WANs, if network security applications or devices would otherwise block it.
2. This isn’t as simple as looking at a phone call, or even a single HTTP “conversation”, in which a server responds to client requests over the client-initiated channel. This optimised routing will only occur if the sender’s traffic can reach the other participant over the optimal route – otherwise it falls back to the Lync servers.
3. It’s really useful to monitor this traffic from both clients, if trying to figure out what’s going on, because they may be doing different things.
4. All of this changes once you have more than two participants.

In my next post I’ll dive a bit deeper technically and step through how we monitored this behaviour. I’ll also reveal some important exceptions to what I’ve described here, which may throw a wrench or two in the works for bandwidth calculations.

For the sake of completeness, I should note that I’m not certain when Lync/Communicator started behaving this way, although for the purposes of setting the scene for my next post, I’m basically just concerned with Lync and the Office 365 context.

On account of this major preoccupation, I would like to get lossless recordings of my own tracks on my media player. When I purchased the Samsung Omnia 7, I was pleased to see it supported .wav files. Since then, I’ve also purchased a book called Mixing Audio. Being a relatively new Kindle owner, I was all about getting the Kindle version of this book, but unfortunately the digital version doesn’t come with ~4GB of .wav files that are referenced routinely throughout the text. So I bought the hard copy. This post recounts my experiences trying to get these 2100+ .wav files on to my Windows Phone, as it was less than straight-forward and very time consuming.

The Omnia 7 does not have a MicroSD slot in Europe (in America, this model is known as the Samsung Focus, and it does), so the only way you can transfer files on to the phone is via Zune (or at least, I’m unaware of any alternatives). Unfortunately, Zune doesn’t support .wav files, even though the phone does, so I’m forced to settle for Zune’s only high-quality format, lossless WMA.

You might wonder why I’m not transferring the files as mp3. While in most cases I would be perfectly happy to do so, two cases where I’m not so happy is with my own recordings and mixing examples like these, where you’re scrutinising the full audio spectrum. Compression would completely alter the reference these files are intended for.

Transcoding Tools

So the problem has become converting .WAV files to .WMA Lossless. Easy enough, right? Wrong. Windows Media Player will do this, but only if the files have been burnt to CD/DVD as audio. In this case, I have my own recordings and a DVD with .WAV files as data, so Windows Media Player is no help.

My next stop was Foobar, my media player of preference. Foobar has a number of conversions tools OOTB, including FLAC and Ogg Vorbis, but no WMA. There are options to create custom converters, which is actually how I stumbled on to my solution. I found this older Hydrogen Audio post which details the process of setting up a WMA converter in Foobar. I tried to do this but the errors I was getting in Expression Encoder that I’ll detail later interfered with the process, so I cut Foobar out of the picture. As I mentioned, I did follow the links in that post to find Expressions Encoder 4, the current version of the only other tool that I’m aware of for transcoding from WAV to WMA. So I installed the .NET Framework version 4 (a pre-requisite) and Expression Encoder, and tried to figure out how to use it. This older post details some of the relevant settings.

Expressions Encoder

The first problem I encountered with Expressions Encoder was that the Import function only allows selection of files in a single folder. This was way too limited for my needs, so I had to use Search Companion to find all .wav files in the parent directory (search for *.wav, including subfolders in the Advanced options) and then drag/drop the files in to Expression Encoder, rather than using the Import function. Unfortunately, Expression Encoder couldn’t seem to handle more than ~600 files at a time before it would crash. It would display the error I mention below, but it appears that the application can’t cope with that many errors popping up at once.

OutputFormat.VideoProfile: “Unable to set properties on null VideoProfile”

Once I figured out I could only convert about 5-600 files in one go, I had to contend with this OutputFormat.VideoProfile error. It popped up for every file that I tried to convert. If converting two-three files it was easy enough to press “Ignore” that many times. Not so for hundreds. After a fair amount of trial and error I found that I could kill Expressions Encoder in Task Manager and launch the “recovered job” when I re-launched the application, and this would bypass the errors. I assume there’s a way to set things up in Expressions Encoder to avoid this problem, but I really don’t have the time to learn how to use it properly given that I’m already wasting hours on this.

Found a recovered job ‘Untitled’ – Click here to open it

Once the “recovered job” is re-launched, it takes a few minutes for Expressions Encoder to analyse the files. On my three-year old Centrino Duo it took about 15 minutes to analyse ~500 files. Once they’ve been analysed, you can specify the WMA Lossless format, language and output properties. The actual encoding process is really quick once all of this is done. I think it may be quicker for lossless than for heavily-compressed files.

The Encode tab on the right is used for specifying the transcoding parameters

Note: you can encode WMA Lossless in up to 24 bits at 96 KHz. Also note that Expressions Encoder was not able to recognise 32-bit source files in my testing, although this might be fixable with the right codec. I worked around this by re-encoding my 32-bit original recordings as 24-bit in Audacity, then transcoding these files in Expressions.

Back to Zune

Once the files have been transcoded, they need to be added to the Zune Collection and the phone needs to synchronise. My next challenge will be figuring out why only ~650 of the 2100+ files initially appeared in my Zune collection (on the desktop application), although I could see all of them getting added. The next day there were 903. Unfortunately, the desktop application won’t synchronise the remainder until it knows they exist, so I suspect I’m now in for some further fiddling, but the lossless audio issue appears to have a viable work-around, even if it’s incredibly inelegant.

Oh, and to end on a positive note, the lossless files sound great on my phone!

As you will have surmised from the title, we were deploying the Office Web Apps. In this case they were being installed with a new SharePoint Server 2010 Standard farm. This is a list of considerations that cropped up during the engagement and a few other bits I’ve picked up since RTM:

• WAN Acceleration: The Office Web Apps help performance over the WAN. This hadn’t occurred to me until I read the recently-released TechNet guidance on geographically-dispersed environments, but this all makes sense, because the documents load progressively. This is explained in more detail in the article. It’s worth keeping this in mind as an aid for global deployments and worth further taking note of the licensing concerns below if the Office Web Apps will be deployed for this reason.
• Licensing: The Office Web Apps license model is based on the volume license for the Office 2010 client. These licenses are supplementary to any SharePoint license concerns, although you might choose to install both in the same farm. This Infoworld article explains it better than anything else I’ve read. The major implications of this are:
  • Assuming this Infoworld article is right, and if the farm will be accessed by external users, they will not be covered by a SharePoint FIS license. All of those users will need to have an Office 2010 volume license (not a home license) in order to access these documents in the browser.
    • This license could be provided by the consuming business or whoever, but it would need to be in place. How this would be monitored, what is and isn’t acceptable use and how licensing audits would work in this scenario are all issues that would need to be inspected in some detail with a licensing specialist.
    • Presumably you could deploy a web application that’s disconnected from the Office Web Apps service applications in order to maintain compliance, but then you negate the WAN optimisation benefits mentioned above, and this clearly has broad IA implications. I’d also recommend confirming the legitimacy of this approach for compliance.
  • On the other hand, internal users are covered by an Office 2010 volume license if it has been purchased, even if the software isn’t installed on their machines. You might find this is the case with enterprise agreements. This means internal users can start to take advantage of the Office Web Apps even if an Office 2010 upgrade is months or years in the future.
  • Talk to a licensing specialist to confirm all of this as it pertains to your deployments, as there are surely other wrinkles I’m not covering and I’m basing this on an article written by a journalist rather than Microsoft or a licensing specialist. The only reason I’m referencing that article is that it’s the clearest explanation I can find.
• Install Media: Once you’ve got the licenses in place, you may have some trouble finding the install media. Even though the media is for 64-bit systems, it’s downloaded from 32-bit downloads on the volume license site.
• Caching: The Office Web Apps Cache is a site collection that should be moved to it’s own database for each web application (in most cases), as the default cache size is 100GB. One way or the other, it’s a different type of content and it makes sense not to clutter actual content databases with this cache data. There are probably different database backup/restore requirements as well. If the existence of this site collection is news to you, read those links above before going any further.
• Topology: Make sure to plan your topology for the Office Web Apps. You will be adding three new service applications and they will tax your system differently based on how they are used. This is a big topic and the planning guidance is fantastic, so I’ll merely point out the Estimate performance and capacity requirements for Office Web Apps document and leave it at that.
• Database Permissions and Application Pools: The Office Web Apps have some quite unexpected database permission requirements. From the Deploy Office Web Apps TechNet guidance:

  Note: You can choose to create a new application pool to be used with a service application. When creating a new application pool, you can specify the security account used by the application pool to be a predefined Network Service account, or you can specify a managed account. The account must have read\write privileges for the SPContent database and SPConfig database. For more information about services account permissions in SharePoint, see Account permissions and security settings (SharePoint Server 2010).
  
  This blows a few holes in the least-privileged model. In this case, we chose to create a new application pool for the Office Web Apps service applications and ran it under the farm account, since the permission requirements are so pervasive. This could be its own separate identity, but the important thing for me is that using the farm account for these services contains the extensive privileges more than granting these wider permissions to the Service Applications application pool would. One way or the other, we’re looking at a different application pool for security reasons. And if anyone’s curious if this is really necessary, it is. You will be unable to actually use the Office Web Apps without these database permissions in an otherwise-least-privileged configuration.

There are a number of other topics to consider before deploying the Office Web Apps, such as the default open behaviour, Office version requirements, browser support and other topics from the solutions world that I daren’t venture in to, but hopefully these infrastructure considerations won’t be overlooked when focusing on those issues.

First Page Load After IIS Reset in SharePoint 2010

One of the key performance indicators I’m measuring is first page load after an IIS reset.

Why not just do an Application Pool Recycle?
Before going any further, I acknowledge that most developers will be able to save a lot of time by recycling application pools rather than resetting IIS – but there are still scenarios when a full IIS reset is required and we’re finding that first page load after an IIS reset is a great deal slower in SharePoint 2010 than it was in 2007. First page load has always been notably slow and people have written warm-up scripts to address this scenario post-reboot, but in SharePoint 2010 I’m noticing speeds are two or three times slower.

I’d initially hoped that I could use this long first page load time to my advantage, which partially explains the time I’ve spent working on this issue here. I was thinking, surely if it takes so long, that extra time will give me a more accurate measure of these performance indicators across different systems. However, as I started to test on server class hardware I was finding that the performance gains were by no means linear and much less than I would have expected. This also held true with i7 laptops, i7 desktops and Amazon EC2. Interestingly, it appeared that the CPU was in no way fully utilised on any of these systems when loading the page for the first time, and these timings did not improve by adding additional CPUs. Earlier tests suggested that disk speed was not a significant factor in first page load times and memory is in no way constrained during these tests.

The Speedy (but evil) White Wizard

In the second instance, we noticed that not all farms were as slow as most of them seemed to be. We stumbled across this accidentally when testing performance in Amazon Web Services (AWS). A colleague did our initial AWS work and we were both very impressed by the initial performance results. A few days later I joined in the fun and built my first single-server instance. We immediately noticed that my first page load times were approximately double the times that my colleague was seeing. Eventually we identified that he used the Farm Configuration Wizard while I had manually created a separate Application Pool for each of my Service Applications. This warranted further investigation.

A note about application pooling
My approach to creating a separate Application Pool for each Service Application is to some extent a hang-over from SharePoint 2007 least-privileged thinking. I was aware that this approach exceeded recommended Application Pool capacity limits, but I didn’t let this trouble me too much based on the single-user load; I’ve always prioritised adherence to the least-privileged model over minor performance degradation. However, based on these seemingly significant performance results, emerging community consensus and the best guidance available today, I decided to reconsider this approach.

A note about AWS
There are a number of broader architectural challenges to conquer when designing a SharePoint 2010 development environment in AWS, which is a topic that I hope to return to in a later post.

A note about the Farm Configuration Wizard
This is the page that greets you immediately after installing SharePoint. It takes care of a lot of Services and Service Applications in one go, but it does some pretty undesirable things as well. In short, for all but the most playful of applications, it’s not appropriate. Build the Service Applications properly.

Reconsidering Application Pooling

As mentioned above, my next step was to quantify the improvements that can be gained through pooling applications. My first test was to delete all of my Service Applications and re-create them in a single application pool. I also deleted all of the web applications and created them in a single, separate Application Pool.

Following my normal development environment build process, I created all of the Service Applications and the Web Applications before tackling the User Profile Service Application. Out of curiosity, I quickly tested first page load times and was happily surprised to find that they had been cut in half. So I took a snapshot and created the User Profile Service Application.

I Blame the User Profile Service Application

After creating the new User Profile Service Application and running an IISRESET, my first page load of Central Administration was almost exactly as slow as it had been with all the Service Applications in their own pools. This was before creating a synchronisation or doing anything with the newly created Service Application. It was merely provisioned. At my wits end, I called it a night.

Having thought about it some the next morning, I decided to create a new web application with a Blank root Site Collection. I already had a similar web application in my farm but I made one key configuration change to the new one. When creating the web application I created a custom Application Proxy Group and removed the Service Connection to the User Profile Service Application. I then tested first page load times on my two blank sites. The new site without the Service Connection to the User Profile Service Application loaded as quickly as the sites did before I created the User Profile Service Application. The original site loaded in the same time as the old sites. The disconnected site was approximately twice as fast to load.

Validating the Results

After reaching this provisional finding, I fired up the Microsoft Information Worker Demo VM. I wanted to test this on a completely different virtual machine (but on the same hardware). I created two new web applications with two new Blank root site collections. I ommitted the Service Connection to the User Profile Service Application on the second web application again. My timings were nearly identical to the timings on the first machines.

Next, I reverted to the earlier snapshot of my development environment – the one with each Service Application in a different Application Pool. I created a new web application with the Blank root Site Collection again and got nearly the same results. In this case, all of the results were slightly slower (a couple of seconds) than they were in my snapshot with all the Service Applications and Web Applications pooled together, but the Service Connection to the User Profile Service Application was a much bigger factor (~20 seconds).

What about the Farm Configuration Wizard Results?
You may be wondering why the sites on the Wizard-configured farm loaded quickly. While I’ve not spent any time revisiting that environment and I’ve never spent much time on servers configured by that wizard, I strongly suspect  this is because the User Profile Synchronisation Service had never been successfully provisioned.

I am still in the process of further validating these results across various hardware configurations and within various virtualisation technologies. My tests should provide better data on the benefits of pooling the Service Applications as well. All of these findings are somewhat provisional, but I’d say the Service Connection results so far are the clearest findings I’ve got to date, by a considerable margin. In short, I think you can expect first page load times to be at least twice as quick when the Web Application is disconnected from the User Profile Service Application.

But I Kind of Need That Service Connection

Touché! You often will. In fact, let me back-track and say that I haven’t really considered how these findings can be applied in the real world yet. In previous development environment iterations, we found that we needed to abandon development in a Workgroup so we could connect to the User Profile Service Application. It may be that some web applications can live without this connection (for instance, many WCM apps), but as I say, my head is deep in performance considerations at the moment and I really haven’t had time to consider these implications yet. However, I will try to revisit the topic reasonably soon and I welcome comments! One way or the other, it’s good to have a better understanding of why 2010 first page load times are so much slower than 2007.

Does this slow anything else down?

At this point, I haven’t had a chance to test much else, but I have tested creating a new web application with and without this Service Connection. No impact. I also tried creating a Publishing Portal within those web applications, and again, no impact.

If you’re curious about the actual performance figures, I hope to publish them in the next couple of weeks. To give a high-level indication, in one environment the connected First Page Load times were ~35 seconds and the disconnected times were ~17 seconds. In slower environments this difference may be even greater.

Fast-forward to the other day and my wife decided to use the Office Web Apps to draft a document rather than using Microsoft Works or Open Office (the other options on her home laptop). Despite some slow responses periodically, all seemed to work well, or so she thought until she got in to work the next day and opened up a blank document.

Cutting to the chase, Word does not auto-save. Typically I think most people (my wife included, of course) would save as they go, but there’s a big inconsistency in the user experience of the Office Web Apps: there is no Save button in Excel, OneNote or PowerPoint. In fact, there’s a Where’s the Save Button? dialogue where you’d expect to find it.

On the other hand, Word not only has the Save button, but it doesn’t auto-save, as the full version of Word does. Note the Save icon also appears beside the Undo and Redo buttons above the Backstage.

In Microsoft’s defence, if you close the browser or navigate away from a Word document, the browser prompts to save changes, but this is a generic form field message.

It turns out this was noted by the Register back at launch.

There are many inconsistencies. The buttons for opening a document in desktop Office depend on an ActiveX control or a Firefox plug-in that does not work in Firefox with Office 2007 or 2003 and only works on Windows. Another annoyance is Microsoft disables the ability to create documents in the browser if desktop Office is available, though you can get round this by using Chrome. A further inconsistency is that PowerPoint, Excel, and OneNote save on the fly, but Word makes you click Save.

Given that the Auto-save functionality in PowerPoint and Excel are sophisticated enough to capture unsaved data even if navigating away from the document or closing the browser, it’s not clear to me why some kind of auto-save or warning couldn’t be built in to Word rather than relying on the browser to display a generic message. Ultimately I suppose this can be handled as a training issue, but a consistent user experience would be preferable, especially if this is targeted at home users.

A few weeks later I was demonstrating these approaches when it was pointed out that we were getting lots of 304 responses. They occurred with each request for a previously-downloaded BLOB Cached asset (more detail added below). Basically, I overlooked the max-age attribute in the BLOB Cache web.config settings. By default, this attribute isn’t present in the web.config file and I simply missed it. Adding this attribute eliminated the 304 results and the caching configuration was complete. Or so we thought.

Edit to provide more detail on the 304 status and Max-Age
A 304 response is a File Not Modified status (not an error), in this case indicating that the browser is making (potentially) surplus checks for each previously-downloaded BLOB Cached file. The max-age attribute gives the file a lifetime in the client’s browser cache in order to reduce these update checks. To be clear, the BLOB Cache stores large objects on web servers to reduce database traffic, but those objects can be served with a max-age attribute that will determine the object’s lifetime in the client’s browser cache. A max-age value of “14400″ means that browsers will cache the file for four hours before checking for an update. This means that updates to BLOB Cached content may become stale if this value is set too high. A common value would be “86400″ (24 hours) but we were satisfied with the balance at four hours. In our case, making this update has not yielded a perceptible increase in performance with the current levels of traffic, but it’s the sort of thing you want to set appropriately in order to optimise things and to allow the environment to scale.

So… yesterday we were told that the 304 results were still appearing after making this change some time ago. When I started testing it I quickly confirmed there were loads of 304 results when I requested BLOB Cached assets for a second time. I tried clearing my browser caches, closed down all browsers and tested in multiple browsers but couldn’t get to the bottom of it, as the BLOB Cache max-age attribute was definitely set in the web.config file.

A bit stumped, I asked knowledgeable people for help. No luck. To recap, this was the full context:

• The Caching column in Fiddler revealed that the max-age attribute was set. I could see the specified value coming through as “public, max-age=14400″. If I changed the value to 86400, the value would get updated on refresh. Clearly the web.config attributes were served correctly.
• In IE I noticed the “Pragma: no-cache” header. In Firefox I saw “Cache-Control: max-age=0″. I couldn’t figure out where these headers were coming from and suspected that was the source of the problem.

Eventually I pinpointed that this behaviour only occurred when I hit F5 or pressed Refresh in the browser. If I typed in the address or clicked on a link I would just get the HTTP 200 results as I would expect. If I hit CRTL+F5, everything would get downloaded afresh (lots of 200s), also as expected. I asked Chris O’Brien if he had any ideas why this would only happen with F5/Refresh and he pointed out that this is exactly what does happen, and a lot of people don’t realise it. Excellent! Learned something new today. Either max-age or F5 could be the cause of the 304 results, but F5 gives a false impression of what users would see when accessing an asset for the second time, unless those users are pressing Refresh a lot. If they’re browsing normally, they’ll in fact get a 200 result. If max-age isn’t set, the 304 results will accurately reveal that users are checking to see if the BLOB Cached file is updated every time they try to access it. In some cases this will be desirable, but it definitely adds up quickly.

Pressing on, I wanted to get to the bottom of what else might be different with F5/Refresh and figure out what was up with those Pragma/Cache-Control headers. It turns out, it’s the F5/Refresh itself that generates the Pragma/Cache-Control header. All mysteries solved! Many thanks to Chris for his help with this. Hope this can help someone else.

Installing the SEO Toolkit

Although Remote Server Administration Tools can be installed on Windows Vista and Windows 7, I have produced the directions below on my Windows Server 2008 R2 desktop. The instructions should be fundamentally the same for any OS once IIS Manager is available locally, however it is installed. To be crystal clear, the SEO Toolkit can be used by anyone with Windows Vista, Windows 7, Windows Server 2008 or Windows Server 2008 R2. It is not a requirement to have access to the web server and it is not necessary to install IIS locally.

On Windows Server 2008 and 2008 R2 the IIS Manager Feature can be added through Server Manager, even if the IIS Server Role is not installed.

Expand the Remote Server Administration Tools node and select the Web Server (IIS) Tools node.

Click Next and Install. Wait for the Add Features Wizard to complete and then download and install the SEO Toolkit. IIS Manager is available in the Administrative Tools menu and should look something like this when you click on your local machine’s connection in the left-hand pane.

The SEO Toolkit home page looks like this.

I shan’t go over everything here because there is an excellent three minute video on the SEO Toolkit home page (linked above), which details the basic functionality of the tool.

Analysing a SharePoint 2010 Publishing site

Click the first link on the SEO Toolkit landing page and Create a new analysis of your site.

The analysis takes you directly to the Site Analysis Report.

Clearly, the Violations are of interest. What sort of things do they tell us? I’ll look at the errors first.

Drilling in to The title is missing, we find that the five pages are links to authenticated content on the out of the box Publishing site template’s home page, which are 401 unauthenticated errors in this case, since this is an anonymous access zone.

The Publishing Portal home page, which includes links to authenticated content

The individual violations

So these errors are unsurprising, now that we know what they are. If this were real content, ideally we would modify it to remove these links to authenticated pages.

So what about that Canonical Formats message?

This violation tells us that a single object can be accessed using two different links. In this case we have two sets of duplicated images. The first two .png files are transparent spacers and the second two are orange.

That’s precisely the sort of thing we would hope to find out about and correct. So what about “The page contains broken hyperlinks”?

Again, all of these links are broken because they point to authenticated content. It’s the same story for “The URL for the hyperlink is broken”, except for the five .gif files that appear there.

In this case the images are actually missing. Again, this is exactly the sort of thing we want to know.

Unsurprisingly all of these errors point out fundamental problems in the site which content owners would want to correct even if they were unconcerned with page rank. The warnings, which I skipped over earlier, provide supplementary insight in to changes that can improve page rank in an otherwise functionally-correct website. Rather than discussing those individually, I’ll just include a screen shot of the Violations tab in the bottom half of the details pane. This tab summarises all of the violations on a selected page, which will improve the editor’s experience when making these changes.

What’s particularly useful about this view is that it is now enumerating each page that violates the specified rule in the top pane, but the Violations tab enumerates all of the violations for the selected object in the top pane (the default home page in this instance).

Some of the other dashboards reveal slow-performing pages, most linked pages, redirects, pages blocked by robots.txt, a status code summary, a list of external links and more. It’s a very useful set of tools. If this has been at all interesting, it’s definitely worth reviewing the video and other resources up at IIS.NET.

Server tools

Everything that I’ve discussed so far can be run against any site that the SEO Toolkit user can browse to. Server access is completely unnecessary. However, there are two added tools that have to be run on the server. This should not be hugely problematic for the content owner, as these tools need to be updated relatively infrequently once they’ve been set up initially.

Creating a Sitemap

Hopping on to my server, I click the Create a new sitemap link and specify the Sitemap file name.

This is my favourite bit. When the Add URLs dialogue first launches, it displays the IIS site files in inetpub.

Not very useful for a SharePoint site, is it? But if we Run new Site Analysis from the URL structure drop-down…

The New Analysis box pops up.

After about 40 seconds of analysis (in my environment) we get the SharePoint site map!

At the bottom of the dialogue the Change Frequency tells search engines how often pages are likely to change. The priority details how important we consider our site to be relative to other URLs on our site (I must confess I don’t fully understand how this works, but I’m not responsible for content. ). You can also tell Search Engines how to identify the Last Modified Date. More information on all of this can be found on TechNet.

One last thing before moving on. We need to add the Sitemap to our robots.txt file. There’s a handy link in the Actions pane to do so.

Also note, in the Related Features on the Actions pane, there’s a link to Robots Exclusion, which brings us to the final tool.

Robots.txt file management

If we click the Robots Exclusion link or the View existing rules link from the SEO Toolkit landing page, we can see that our sitemap.xml file is being referenced, as added above.

This is confirmed if I click the Open Robots.txt link in the Actions pane.

If we click Add Disallow Rules or Add Allow Rules we get a similar dialogue, and in both cases we will want to specify our previous Site Analysis for the URL structure.

Now if I want to exclude all of the content that I don’t want to index I can just tick the appropriate boxes from my last analysis (note that this looks different than the selectable options from the Sitemap dialogue).

Voila! Paths are excluded and the robots.txt file is updated.

One thing to note is that the robots.txt file did not appear for me during testing until after I stopped and started the website in IIS. This is only an issue when it’s created for the first time, but worth noting. I believe this is also true for the sitemap.xml. file.

So… these are good tools! Once configured, the robots.txt shouldn’t need to be updated often and the web managers should become aware of any problems soon after they occur through their own use of the reporting tool. In short, these tools devolve a great deal of control and insight and there seems to be very little reason not to use them.

We have also experimented with generating the server side outputs using PowerShell, which a colleague of mine will detail soon and I will post here when ready. If there is any reluctance to use this IIS.NET extension in production infrastructure, a combination of PowerShell for file generation/management and the SEO Toolkit for reporting may be a sensible solution.

Issues

Guest user accounts

Guest virtual machines have been configured in a workgroup in order to conserve resources that would be spent on domain services. Additionally, developing on a domain controller is less than ideal for a number of reasons including performance tuning, administrative complexity, start-up times and security.

I created the development virtual machine with 160 local user accounts that have been logged on to the portal and the MySite application in order to create a basic profile. If there is a need to script creation of local user accounts, the net user command will be useful. However, this will be of limited assistance for complex profile requirements, since there is no way to synchronise with a directory and since the local users have no associated profile data, but it may be helpful for testing or demonstration.

If LDAP user accounts or other directory objects are required for development purposes (user profiles for instance), consider using Active Directory Lightweight Directory Services. This is the successor to Active Directory Application Mode (ADAM) in Windows Server 2003. It is a Windows LDAP directory that supports user and group objects without a full-blown domain infrastructure.

There will be some scenarios when a full domain services infrastructure is required for development. In those cases it may be preferable to run a second virtual machine as a domain controller.

Hibernate and Sleep

Hibernate and Sleep are disabled automatically when Hyper-V is installed. This is by design. Hyper-V disables this functionality, as the guest virtual machines could be damaged by a Hibernate or Sleep operation in the host if they were not saved gracefully. If, on the other hand, all virtual machines had to be put in to a saved state before a host machine could be put to sleep or hibernated, this would mean extending the wait time for these operations to unacceptable levels, as they are also automatically triggered by low battery warnings. Unfortunately we need to live with this behaviour.

Do not travel with a running laptop

Putting a running laptop in a bag will cause it to overheat quickly and is likely to damage hardware.

Improvements to Start-up and Shutdown times

These builds should start up and shut down in less than two minutes (closer to 90 seconds). Keep in mind that virtual machines can be safely saved and work can be resumed quickly when the machine is restarted. Since all of the development work will be taking place inside the virtual machine, this should reduce the Hibernate/Sleep annoyance.

Virtual PC won’t run on Windows Server 2008 R2

Windows Virtual PC will not work on Windows Server 2008 R2, as it was designed specifically for Windows 7. Earlier versions of Virtual PC may install on Windows Server 2008 R2, but they will not co-exist with Hyper-V, so do not install them.

Hyper-V role won’t work after SysPrep

This shouldn’t be an issue, as we have set up automated deployment, but it’s worth noting that this is a known issue. There are time-consuming work-arounds to fix some of the problems that this will cause, but they are best considered as a last resort.

Colour management

Colours are limited to 16-bit in Hyper-V guests. If a fuller spectrum is required, it should be possible to test in full colour in a browser on the host.

Resolving host names from an internal domain

During our pilot we identified that fully-qualified domain names resolved successfully but host names would not resolve without the full domain name. To satisfy this requirement we have added our internal DNS suffixes to the ICS Connection inside the development virtual machine.

Manually adding DNS suffixes

If a network adapter in a guest virtual machine loses these settings by deletion/re-creation of the adapter, or for some other reason, the setting can be re-entered as follows:

• Go to the IPv4 properties on the ICS Connection and select Advanced.
• On the DNS tab select the Append these DNS suffixes (in order) radio button.
  • Add internal.domainname.local and other.domainname.com.
• Un-tick the DNS registration box.
• Select OK, OK and Close.
• Make sure that this change is captured in all snapshots as necessary.

Internet Explorer (64-bit version)

Adobe flash player does not currently support 64-bit browsers. You’ll have to use the 32-bit IE or another browser if you want to view flash files. We recommend using the 32-bit version by default.

Hyper-V Manager UAC prompt work-around

If the UAC prompt on Hyper-V Manager is annoying, try launching Server Manager and navigating to Hyper-V in the Roles node. This has the added benefit of exposing the Hyper-V event log messages and service states in that top Hyper-V node. These are not visible in Hyper-V Manager. Awareness of these messages and the service statuses will help to resolve Hyper-V issues faster.

Test DVD burning

In our pilot we identified that the DVD burner drivers don’t work for burning in Windows Server 2008 R2 on a Dell XPS M1330. This was also true on a Lenovo laptop. Chipset updates, driver updates and a Microsoft KB registry hack all failed to make a difference. The Matshita (Panasonic subsidiary) site does not support the products directly (they point to the laptop manufacturer). Dell and Lenovo had not released new drivers when we launched. As DVD burning has changed in Windows Server 2008 R2 this may have a wider impact.

Bluetooth doesn’t work

The Bluetooth stack is missing from Windows Server 2008 and Windows Server 2008 R2. In Windows Server 2008 there were fairly elaborate means of porting the stack from Vista, but results appear to be spotty at best.

WorkItemTypeDeniedOrNotExistException when trying to open work items

This error occurred in the first release of our guest build because I installed Visual Studio 2008 SP1 before the Team Foundation Client (TFC), so the TFC did not get upgraded. The fix is to un/re-install Visual Studio 2008 SP1, or to make sure that the TFC is installed before Visual Studio 2008 SP1.

NUMA nodes and RAM allocation

It is important to not exceed NUMA node limits when assigning RAM to virtual machines, although this will not apply to many laptops, as most will have an SMP architecture. It is beyond the scope of this post to go in to NUMA nodes in great detail (and in truth, my understanding of it does not reach beyond a few hours of research), but the limits in your environment should be understood so that performance does not suffer. As a starting point it’s worth confirming the type of CPU architecture and looking at this in more detail if it is NUMA. The performance and capacity requirements for Hyper-V document on TechNet explains this well:

Configure the correct amount of memory for Hyper-V guests. During the testing, no change had a greater impact on performance than modifying the amount of RAM allocated to an individual Hyper-V image. Because memory configuration is hardware-specific, you need to test and optimize memory configuration for the hardware you use for Hyper-V.

The initial goal of the testing was to make the Hyper-V image as similar as possible to the physical hardware image against which it was being compared. Based on that goal, the Hyper-V images were originally allocated 32 gigabytes (GB) of RAM, which was the same amount of RAM as was on the physical servers being tested. However, the initial test results showed that with that configuration, the Hyper-V images could sustain a load that was only about 70 percent of the load on the physical hardware. After investigating the Event Viewer on the Hyper-V host machine in the Windows Server 2008  Custom Views, Server Roles, Hyper-V Events, it was discovered that the RAM for the Hyper-V images was being spread across multiple non-uniform memory access NUMA nodes. This information confirmed that performance declined when memory was allocated across nodes. After trying different configurations it was determined that for the hardware being used, 8 GB of RAM was the maximum that could be allocated to a Hyper-V image without crossing NUMA nodes.

To reiterate, this means that in Microsoft’s tests, Hyper-V performed significantly worse with 32GB allocated to a virtual machine than it did with an 8GB allocation. The exact size of the NUMA node boundary will vary by vendor, so make certain to gain an understanding of the number of nodes in your system. Divide the total RAM by the number of nodes in order to find the memory limits of a virtual machine. This does not mean that additional virtual machines can’t be run beyond a NUMA node boundary, if there is sufficient RAM available. The node boundary is the limit of optimal process performance. Beyond this limit, the virtual machine will suffer from degraded performance because it needs to use memory from an alocal address space.

However, NUMA isn’t the only thing to worry about when finding an optimal RAM allocation. Based on test results during our pilot we could push our virtual machines to up to 2250MB RAM, depending on the amount of activity in the host. In some cases it may be possible to get up to 2500MB RAM for a virtual machine on a 4GB RAM system, but this was not consistently achievable in our tests. If it’s necessary to achieve that, the virtual machines should be started up soon after booting and before any major client application activity is started on the host machine. Client application activity should be kept to a minimum when allocating this much RAM to virtual machines. We also found that host performance was often reduced to an intolerable level whenever there was less than 2GB RAM available to the host for an extended period of time. 1.75GB RAM may be achievable, but this should be tested extensively for your needs.

Additionally, saving a virtual machine’s state becomes risky when there is less than 2GB available to the host, as the machine will not resume from the saved state if there is insufficient resource available to it.

Periodic but routine loss of connectivity on the host machine

As I’ve been tracking here, we’ve documented repeat problems with periodic (but routine) loss of connectivity on the host machine. This is still an open issue. More info here:

Routine loss of connectivity on a Hyper-V host’s external connection
More on routine loss of external network connectivity on Hyper-V hosts (not guests)

Hyper-V performance suffers during graphics-intensive operations

This has been covered by Ben Armstrong in considerable detail and I’m continuing to track it:
Hyper-V graphics performance and SharePoint 2010 development
Hyper-V graphics performance is on the way… if you need a new laptop
The definitive word on Hyper-V high-end graphics performance

Aero Glass

Unfortunately, due to the graphics performance issues in Hyper-V mentioned above, there is a significant graphics performance hit when using Aero Glass. This does not slow down overall systems performance, but graphics-heavy operations will suffer in most Hyper-V environments. To this end, we do not recommend installing Aero Glass, but if you want to put it to the test feel free.

How to enable Aero Glass

• Make sure the Desktop Experience is activated
  • On the Dell XPS M1330, make sure BIOS A14 or later is installed
    • Confirm the latest NVIDIA drivers for Windows 7 x64 are installed
    • Turn on the Desktop Window Manager Session Manager service and switch to automatic start
    • Turn on the Themes service and switch to automatic start
    • Switch to an Aero theme

Sidebar

It is also possible to add the Windows 7 Sidebar to the host dekstop. We haven’t tested this extensively enough to provide documentation on the best approach, but we have done it successfully. If there is sufficient interest in this technique I will add a follow-up post in future.

Storage

Be prepared for snapshots to increase the local storage requirements considerably. Some of our developers are legitimately struggling to work on three projects concurrently with 300GB local storage. One option we are considering is eSATA over PCMCIA as a means of increasing total spindle speed and storage but we have yet to begin testing this approach. We are specifically interested in eSATA since Hyper-V does not support system VHDs on USB. If we pursue this option I’ll post the results.

Results

Putting things in perspective

It’s worth repeating that we’re asking this system to do many things it was not intended to do. It is a server operating system with an enterprise virtualisation technology. The Microsoft virtualisation team will tell you that Hyper-V was not designed with developers in mind. To quote Ben Armstrong:

As is being discussed at length here – Hyper-V does not play well with high-end video cards (which are far more common on desktops than servers).  Hyper-V also disables sleep and hibernate, as well as increasing the power utilization of the computer.  All of these things would need to be addressed before we could even consider putting Hyper-V in a desktop product.

In short, we bent this system to development needs because of the strength of the technology, despite these imperfections. There are fundamental compromises that can’t be avoided when using a server operating system as a mobile workstation but we believe that we can deliver SharePoint projects as a team better with this technology than without it.

Hardware

Whether laptops are the ultimate hardware solution is a different can of worms, which I’ve chosen to avoid in this series of posts. I’ve tailored the approach to laptops since that is what we have and the approach can be ported to workstations or shared virtual infrastructure.

The developer experience and the bottom line

There’s no question that using snapshots, import and export in Hyper-V adds a complex tier to the development experience and there will be a learning curve for those who are less familiar with virtualisation or don’t use the advanced features often. However, we have achieved an immediate and measurable gain in stability and environment consistency through the use of standard builds, snapshots and exported project-defined environments.

Conversely, it’s worth keeping in mind that as desirable as standardisation is, there are times when it hinders more than it helps and on those occasions a non-standard build may be more appropriate. Considering alternative builds is a much less cumbersome proposition with  the combination of WDS, Shrink Volume and Dual-booted systems or the new Native boot from VHD. The key consideration to keep in mind is that most other approaches will entail a sacrifice of what I lump together as the “management benefits” of Hyper-V (snapshot, import and export). For instance, you may consider allowing a team to develop on native operating systems for a project, but then a team member may lose a day if they need to rebuild their system, or the support team may need two days to build an environment in Hyper-V later on, or a team member may need to split her time with a team who use Hyper-V for their project, or the original project may fork. Memories of project difficulties gone-by come flooding back. While it’s always worth considering options, if you spend time identifying a standard approach for your business, it’s probably best to stick with it unless there’s a truly compelling reason not to.

Last week we built a Windows Server 2008 R2 Hyper-V host that we used for our SharePoint 2010 launch event at Microsoft London. We were practicing the demonstration from connected Windows 7 laptops but we also wanted to understand the limitations of presenting from the host itself, should something go wrong with the networking. For the most part this worked fine, but we ran in to problems when we tried to run the media web part. The dialogue box would pop up as normal, but there was no option to Play. If I clicked View I would be prompted to download the .WMV file. At this point I realised that this hardened server did not have the Desktop Experience enabled, so it didn’t have a default media player.

After installing the Desktop Experience and rebooting (note: this does a few reboots), the pop-up looked the same; the Play button was still missing. Now if I clicked View it would launch in Windows Media Player.

Eventually it occurred to me that the server might be missing Silverlight. We install it by default, so it took some time for me to catch on. I checked the installed updates and sure enough, it was missing. When I tried to run Windows Update it revealed that Silverlight was the only availably update. I tried to install it and I quickly got error 80244019. Searching for a solution suggested a few possible answers – mostly network/DNS-related. To expedite things I visited the Silverlight site and installed it manually. I believe the problem may have been DNS or proxy-related but since all other Windows updates installed fine it’s a bit of a mystery.

Installing Silverlight manually did the trick. The Play option returned in the pop-up and video played normally. Out of curiosity I removed the Desktop Experience and tested again. Everything continued to work normally. All of this is totally clear in retrospect, but with the new technology cocktail that is SharePoint 2010, Silverlight 3, IE8 and Windows Server 2008 R2, it’s easy enough to loose sight of the obvious.

“Background Removal is a new feature in Word, Excel, PowerPoint and Outlook that makes this process quick and easy for any picture. Unlike similar tools, the Office Background Removal tool doesn’t just select color ranges or trim to a border you draw. Background Removal uses new capabilities and algorithms from the Microsoft Research and Development team in Cambridge, UK to achieve better results automatically with very little effort or fine tuning from the user.”

I thought FoxIt was a good alternative for the first year or so that I was using it, but (like an adolescent) as it matured some things about it started to annoy me and I thought it was worth trying out some of the alternatives on the market. Of the Windows options from the F-Secure recommendation, I didn’t get very far with Okular or Yap but I spent a good deal of time with Sumatra at work and MuPDF with the plugin for Firefox at home. I wasn’t unhappy with either of these options, as my PDF reader requirements are very basic. I love that Sumatra can be run without installing anything and is small enough to carry around on a memory stick. No more PDF readers on servers… However, after inflicting Sumatra on some unwitting recipients, I decided to find a heftier alternative, as they need more than up, down and zoom.

Enter: DocuTrack’s PDF-XChange Viewer. It’s got loads of features, is reasonably lightweight and does the job in every way that I’ve put it to the test. The only thing that irritates me is the size and number of toolbars running by default, but they’re easy enough to turn off. Admittedly my needs are few, but I’ve not had any other complaints about it since pushing it to about 20 other people. If you fancy taking the plunge and take me up on any of these suggestions, I’d love to hear feedback, as I’m discovering that it’s nearly as divisive as the media player question. I <3 Foobar, for the record.