Sunday, June 17, 2012

Troubleshooting SharePoint 2010 Search

You’ll Need a Logger For That

Although it’s a killer feature, Search is as awkward as ever to troubleshoot inSearch SharePoint 2010. The Event Viewer chatter is pretty meek for troubleshooting, and the Search Center crawler reports have couple minute lag which makes them awkward to work with. The ULS logs can be great, but some troubleshoots produce so much data it’s hard to get a feel for everything that’s happening reading static flat files.

Case in point, other users on the web have gone so far as to set up fiddler as a reverse proxy, just to get a feel for what’s happening to with Search as it goes about its crawling business (it’s cool to watch btw).

Enter the aptly named real time ULS log viewer “SharePoint LogViewer”.

It’s the most sane conversation you’ll ever have with this service. You’ll be able to see everything from the callbacks from the administration site to the noise the crawler itself makes, all in real time.

You should recognize it all:

  • Listing Content Sources
  • Authenticating with the individual web applications
  • Web Service calls to actually get list items
  • And a lot of other minutia you might be interested in…

What’s more, you can set filters on the chatter to help spot pertinent info (see figure 1). It’s usually enough just to start the logger, begin real time logging (takes a couple seconds to connect to farm), and setting a filter (say “Search”) to get all the info you need to start discerning what’s going on with one of your search crawls.


Figure 1: SharePoint ULS mid crawl

It’s Great When it Works

Search is one of the coolest features that comes with SharePoint. Its ability to index a wide array of files, provide security trimmed results, and scale to huge 100 million item index sets (just to name a few) have help set it apart from other search alternatives (like Full Text Search, Lucene, Google Minis, etc…). If you have buckets of money, there’s also a great roadmap into FAST Search which has it’s own impressive additions to help you support even more robust search needs.

As the search space has matured, the SP 2010 Search itself has had to grow to keep up. For developer to continue to be able to troubleshoot/understand how these services are working, it’s also likely that your tool belt need to grow as well. A good real time log viewer is sure to make a great addition.

My Best,

Monday, April 16, 2012

Extending the FedAuth / Claims Auth Ticket in SharePoint 2010

It’s Not Long Enough

For those whom have serviced calls about authentication in SharePoint 2010, a portion of your last two years may have been spent learning the broad range of changes that have come with the Federated and Claims Based models that are now part of authentication in SharePoint 2010.
One common scenario, especially for Forms Based Authentication sites (implemented with CBA) is how to extend the login window.

The Default – 10 Hours?

Once you switch (or initially create) a web application over to CBA and implement a Forms Based Authentication provider you might start to notice that the cookie window has a peculiar lifetime…it’s 10 hours for both Windows and Forms Based Authentication.
It’s worth noting that until you implement the FBA the cookie’s still behave similar to those in Classic authentication (expire at end of session).
The cookie is relatively easy to find (seen here with the Firefox Add-in Cookies Manager+).

Authentication Ticket from FBA Login
Figure 1: Authentication Ticket from FBA Login

How to Extend It

It’s very likely there’s someone in your organization that either doesn’t like logging in, or isn’t very good at it. For these genre of users the new FBA behavior isn’t a boon and it’s only a matter of time before the conversation of extending the login windows comes up.
Strangely enough there’s not a ton of straightforward documentation (partly the reason for this post) or options to remedy this.

One of the easiest ways to extend this is by altering the configuration of the Secure Token Service with PowerShell. To be fair, it has a pretty big impact (farm wide) and introduces some other concerns. One of these is the fact that the user won’t be tasked with authenticating for the duration of the new extended window. While this sounds obvious (that’s the point right?) the fact that users are authenticating less often means that other actions (say in Active Directory or at application levels) like invalidating users or hooking/processing events at authentication time won’t be firing.

If you extend the window to say 30 days, it’s possible that users that you’re marking invalid may have access to systems who don’t check said flag (and focus simply on authorization…I mean that’s partly the raison d'ĂȘtre for CBA) for quite a bit longer than if you’d left it alone.

For sites that we’ve been asked to do this, we usually implement an HttpModule to inspect the user for any properties that are normally screened at authentication time so that we know we’re dealing with a valid user. So far this has seemed to mitigate the effects.

Ok, enough talk. You can see the existing farm level settings for the Secure Token Service with the following PowerShell command:


That should yield the following output, notice that the Forms and Windows Token Lifetime are individual settings (both initially 10 hours) and can be set independently.

Secure Token Service Settings
Figure 2: Secure Token Service Settings
You can change these settings by running the following commands for the Windows and Forms token lifetimes respectively:

Set-SPSecurityTokenServiceConfig –WindowsTokenLifetime [minutes]

Set-SPSecurityTokenServiceConfig –FormsTokenLifetime [minutes]

This is of course done with PowerShell, I haven’t seen a way to do so with the web.config for the service.
Once making the changes, they begin immediately (no need to recycle app pools or web servers). Your cookie should look like below (set to 30 hours as an example).

Figure 3: Extended FedAuth Cookie

I know it’s not ideal (has huge impact and raises other considerations), but it IS the most straightforward way of extending the cookie, and if it’s available, you should at least know about it.
Hope that helps,

Wednesday, March 7, 2012

No SharePoint 2007 Menu Flyouts/DropDowns in Chrome

Broken Navigation Menus

As of Chrome 15 (we think) SharePoint 2007 sites started to behave a little funky. The stock SharePoint menu started to behave erratically, intermittently not showing navigation flyouts. To be clear, on every other page load the navigation would behave appropriately, but the other half of the time the menu wouldn’t do anything on hover, making it impossible to see sub navigation items under a given header.
This seems to affect all SP 2007 sites for Chrome version 15 and later. The error strangely enough seems to be more related to javascript than rendering behavior.

The Fix

I’ll save you the troubleshoot, we added a late loading script that tells the menu that flyouts are indeed kosher. For some reason this variable gets set incorrectly for newer versions of Chrome. There’s a good chance MS may not release a fix for this since Chrome isn’t a supported browser for SharePoint 2007.
<script type="text/javascript">
    //Updates the stock SharePoint menu settings to allow flyouts,
    //seems to get intermittently improperly set on Chrome >= 15.
    flyoutsAllowed = true;
That’s it, pretty simple (especially if you already have some jQuery running around). Otherwise you can get by having the script run really late in the document life cycle.

Hope that Helps Someone,

Wednesday, February 29, 2012

How SharePoint Deals With Time And Time Zones

What’s Going On Here?

It’s a little weird how hard it is to find some detailed descriptions on how SharePoint handles time, and time zones. So here’s a dump on a lot of things time and time zone related to better help you troubleshoot and predict behavior.TimeWarp

How Time Is Stored

Fields like created/modified are stored will all other list item properties in the dbo.AllUserData table as tp_Modified and tp_Created. Like all list items, events are stored in this table as well.

Time Calculation With Respect to Time Zones

Time gets stored using the clock of the machine, this is not influenced by the time zone of the machine. If the clock on the server is off, the time stored is off by the amount of the clock skew. Changing the time zone of the server (OS) doesn’t affect how time is stored. This is true for most applications.


If a server is at 6pm PST (Pacific) and you change the time zone to EST (Eastern) sure the time looks like it changes from 6pm PST to 9pm EST, but those times are equivalent. 9pm EST is just another way of writing 6pm PST. Events taking place in California are really happening at the exact same time in New York, their citizens just happen to format time differently. Time zones are just another way of expressing (formatting) the current time. To be clear. There is only one time, there are many time formats…and there is no spoon.

The Clock IS Important Though

If you change the time on the server (OS) clock you’ll change how time will get stored (for new edits/additions). In this case you’ve actually had an impact on time. Namely because going from 6pm PST to 7pm PST means someone actually lost an hour they’ll never get back (not that fake ones you tell yourself about when you travel through different time zones). By changing the clock from 6pm PST to 7pm PST, you’ve effectively moved the server through time. As a result, this will effect the times that items are recorded to be stored at.

How Time Zones Get Set

By default all your users see the time as it is formatted at the root site (SPWeb) level. These settings get set when the site collection is provisioned by the web application default time zone (Figure 1 shown below). These settings get copied down when new sub sites are provisioned under the root site. Finally, if any user doesn’t like the way dates and times are displayed, they can override how time is presented to them using their personal regional settings (Figure 2 also below). But most of the time its the root site settings time zone that users are complaining about.

Default Site (SPWeb) Time Zones

This setting dictates the formatting/time zone for all data in the site. By default these get copied down from the parent web at time of creation. You can view/change these settings (if you’re a Site Administrator) by going to:
  1. Site Actions->Site Settings
  2. Click on Regional Settings
Site (SPWeb) Regional Settings including Time Zone
Figure 1. Site (SPWeb) Regional Settings
This setting is dictating how time is displayed to all users whom haven’t expressed a preference using their personal regional settings. At the root of the site collection (root site), you can also tell all sub sites to inherit the settings of the root web, resetting them to a single time zone.
Changing the Regional Settings time zone for a web programmatically looks like the code below. Time zone IDs come from a list of known time zones.

using (SPSite siteCollection = new SPSite("http://localhost"))
    using(SPWeb websiteRoot = siteCollection.RootWeb)
        SPRegionalSettings regionalSettings = websiteRoot.RegionalSettings; 
        SPTimeZone timeZoneIndianaEast = regionalSettings.TimeZones[15]; 
        regionalSettings.TimeZone.ID = timeZoneIndianaEast.ID; 

User Site Collection Time Zones

We say user site collection time zones settings because even though this is stored per user, these settings reset in across site collections. Just because you’ve set this in site collection A doesn’t have any impact on site collection B. It will follow you around across many sites (SPWeb) within a site collection though.

You can override the default site time zone by going to:

  1. Click on your username in the header and click on My Settings.Click on My Settings to access Regional Settings
  2. Click on My Regional Settings.
  3. Uncheck Always follow web settings and override the time zone with one of your preference for this site collection.
User Site Collection Time Zone settings overriding the default time zone.

Figure 2. User Site Collection Time Zone settings  
overriding the default time zone

After you click OK, time will be formatted for the time zone you dictate.
You can change this for a given user with the following code:

//Create a new regional settings based off of the current web.
SPRegionalSettings regionalSettings = new SPRegionalSettings(web, true);

//Set some settings
SPTimeZone timeZoneIndiaEast = regionalSettings.TimeZones[15];
regionalSettings.TimeZone.ID = timeZoneIndiaEast.ID;

//Assign to user and update.
user.RegionalSettings = regionalSettings;


There are some interesting effects of the time zone settings. For instance, if you ship a site collection to another time zone, it will still behave as if it’s in your original time zone with no difference at all so long as the clock at the host is the correct time.

Different users can have very different time settings across different site collections. The good news is that they get the final say on how time is formatted.

Both these settings can be manipulated programmatically. When these settings are set appropriately they can help users from very different time zones collaborate on data with time formatted in a way that make sense to them. This especially applies to event/calendar items.

Hope that helps someone, regional settings can have a big impact on collaboration projects.

My Best,

Sunday, February 12, 2012

Troubleshooting The Windows SMTP Server

Why Isn’t My E-mail Getting There?

Email is a very common but hard feature to troubleshoot.
No one gets a hi-five for implementing an email feature. I’ve tried it before, it’s impressively devoid of gratitude.
Email is one of those features that’s just supposed to work and very little thought is often given to how difficult it really is to get right. Even if users didn’t expect it to be instantaneous and scale to legendary sizes, they definitely wouldn’t hand out accolades for all the troubleshooting that email brings to your door.
Email is as complicated as ever, and requires a ton of different systems to be in sync just for a single message to get through. Components like:
  • Application Settings
  • SMTP Server auth/relay settings
  • DNS (appropriate MX, A, and potentially PTR and SPF) both internally and externally
  • Email client and Server server spam filtering (text processing, attachment blocking, black listing, etc…)
…just to name a few, often present issues and ensuring their correct often crosses multiple disciplines and sometimes organizations. The broad domain of knowledge required to diagnose email issues can often cause their troubleshoots to be prolonged.


Fortunately most troubleshoots usually fall into one of three categories:
  1. Application/SMTP Server Issues – These are problems between the application and the SMTP server, usually there’s a stack trace or some kind of very visible error code thrown for the application or in the Event Viewer/SMTP Log files. These are for things like:
    • Unable to authenticate
    • Unable to relay
    • Email is invalid (one of its fields breaks the RFC)
    • etc…there’s a bunch of them, but they they’re usually pretty easy to troubleshoot.
  2. Sending SMTP/The Internet/Destination SMTP – This is where things get a little bananas. A lot of SMTP server’s aren’t great at logging (ahem, all versions of IIS to date), and it’s hard to get feedback on where in all things are going sideways.
This is also where a lot of developers start to feel out of their element. Checking DNS records and relating them to email behavior has traditionally been reserved as a core IT task. This is where tools like SMTPDiag save the day. Weighing in at 79 KB this tool can be a life savor that stops you from needing to memorize telnet “ehlo” commands and other awkward SMTP command syntax.
SMTP Diag runs a variety of tests, like DNS lookups and then goes through the motions of sending (doesn’t actually send) an email for every server that has an MX record. This usually tests:
  • DNS – Both sending and receiving domains have MX records, have A records.
  • Reachability – Both the sending and receiving domains’ email servers are accessible.
  • Accessible - Connects to SMTP servers and goes through the motions of sending an email from/to the given addresses.
Here’s an example of it running in verbose mode:
SmtpDiag.exe “” “” /v
Just today the utility helped me realize that some emails stuck in my Windows 2008 SMTP server queue were really stuck there not because my SMTP server was set up incorrectly, but because my ISP was stopping outgoing port 25 traffic. My only regret is not running the thing earlier in my troubleshoot (I started at my application and SMTP server).
3. Assuming both of the above check out, then you most likely have something on the user’s email client or on the destination email server (read spam filtering). Which also means it’s officially not your fault.
I hope the above helps someone, these troubleshoots can often be a nightmare and there’s nothing like a good tool to help you wake up.
Good Luck,

Wednesday, January 11, 2012

401 SharePoint Designer Connectivity Issues

How Many Times Must I Log In!?

SharePoint Designer 2010 is definitely a huge improvement over 2007. It’s free and has a ton of enhancements to make customizing a SharePoint install easier than ever.
It IS however, just as difficult to troubleshoot as the previous version when things go wrong. This is mainly because almost everything SharePoint Designer 2010 does happens over a web service. This means that a lot of the errors that you tend to get back (if you get any at all) are cryptic, and troubleshoots can be prolonged.
The other day one of our developers was trying to do some edits to a local SharePoint 2010 dev farm via SharePoint Designer 2010 and was having some issues logging in.
The user was a site collection administrator and every time they entered a correct credential, they were prompted to enter it in again (and again).
In addition when running Fiddler to inspect the login traffic, the log was full of authentication attempts to the login service (/_vti_bin/shtml.dll/_vti_rpc) but all returned 401 http status codes (unauthorized). There were no relevant events in the Event Log or in the SharePoint logs (14 hive\Logs directory).
Normally (if accessing it via a host header), we’d assume this has to do with IIS loopback checks, but this wasn’t the case. After a lengthily troubleshoot we found out that the behavior actually had to do with a change we’d made to the web.config.
As it turns out, when you enable ASP.NET tracing, you’ll no longer be able to login via SPD. We had the following line in our web.config.
    <trace enabled="true" requestLimit="40" localOnly="false" />
After we removed this line, we were able to login. We’re still digging around trying to discern how the two are related, but in the interim I’m hoping this saves someone out there some time.
HTH Someone,