Saturday, January 29, 2011

Extending CoreResultsWebPart to Handle Search Queries Written in FAST Query Language

This was one of unanswered items in my recent two presentations on Search at TSPUG and MSPUG, so I was driven to figure it out and eventually did get it to work although not without some controversial steps. In this post I chose to also describe other approaches I tried and things I learned along the way, which didn’t necessarily get me to the end goal but may still be useful in your specific scenario. If you are looking for just extending CoreResultsWebPart so that it can “understand” FQL then you may want to scroll down a bit. You can download the complete source code here. I was testing my code on a SharePoint Server 2010 with December 2010 CU installed.

As you may know search web parts in SharePoint 2010 are no longer sealed, which gives you lots of flexibility via extending them. CoreResultsWebPart is probably the most important of all and therefore it is a great candidate for being extended. I wanted to take a search phrase passed as a query string argument to a search results page and write my own FQL query using this phrase as a parameter. My FQL query would do something interesting with it, for example use XRANK to boost documents created by a particular author. I certainly wanted to leverage all the goodness of CoreResultsWebPart, just use my FQL query instead of a plain search phrase. Contrary to my expectations it turned out to be not trivial to accomplish. So let’s dive into the details.

The story started with it completely not working, so I was forced to write a custom web part that used query object model and in particular the KeywordQuery class to deal with queries written in FAST Query Language (FQL). This is the one I have demonstrated at MSPUG and (with little success) at TSPUG.  Below is a fragment showing how the query is submitted and results are rendered (BTW here is related MSDN reference with example).

using (KeywordQuery q = new KeywordQuery(ssaProxy))
{
q.QueryText = fql;
q.EnableFQL = true;
q.ResultTypes = ResultType.RelevantResults;
ResultTableCollection tables = q.Execute();

if (tables.Count == 0)
{
return;
}

using (ResultTable table = tables[ResultType.RelevantResults])
{
while (table.Read())
{
int ordinal = table.GetOrdinal("Title");
string title = table.GetString(ordinal);
TitleLabel.Controls.Add(
new LiteralControl(
String.Format("<br>{0}</br>\r\n", title)));
}

table.Close();
}
}

As you can see, rendering of results is very basic. Of course it can be made whatever it needs to be but why bother if there is CoreResultsWebPart around, which does it great already and is also highly customizable? So at this point I rolled up my sleeves. I have a book titled “Professional SharePoint 2010 Development” with a red Boeing 767 on the cover, and in Chapter 6 there is a great discussion of how to extend CoreResultsWebPart. Also Todd Carter speaks about it and shows a demo in his excellent video on Search. If you haven’t seen it I highly recommend spending an hour and 16 minutes watching it (Module 7).
Empowered with all this I wrote a web part that extended the CoreResultsWebPart and used a FixedQuery property to set a custom query, being almost one-to-one copy of Todd’s demo example. Here is the listing of this web part class (note how ConfigureDataSourceProperties() method override is used to set the FixedQuery property):

public class ExtendedCoreResultsWebPartWithKeywordSyntax : CoreResultsWebPart
{
protected override void ConfigureDataSourceProperties()
{
const string LongQueryFormat = "(FileExtension=\"doc\" OR FileExtension=\"docx\") AND (Author:\"{0}\")";
const string ShortQuery = "(FileExtension=\"doc\" OR FileExtension=\"docx\")";
string query = null;

if (String.IsNullOrEmpty(AuthorToFilterBy))
{
query = ShortQuery;
}
else
{
query = String.Format(LongQueryFormat, AuthorToFilterBy);
}

this.FixedQuery = query;
base.ConfigureDataSourceProperties();
}

[SPWebCategoryName("Custom"),
Personalizable(PersonalizationScope.Shared),
WebPartStorage(Storage.Shared),
WebBrowsable(true),
WebDisplayName("Author to filter by"),
Description("First and last name of the author to filter results by.")]
public string AuthorToFilterBy { get; set; }
}

The web part actually filters results by a given author. It uses keyword query syntax and not the FQL so we are far from being done yet. Remember how in the previous code fragment there was a line q.EnableFQL = true;? If just we could set it somewhere we would be essentially done! Well right but the KeywordQuery object is not directly accessible from the CoreResultsWebPart because it uses federation object model on top of the query object model (as do other search web parts). Purpose of the federation object model is sending same query to multiple search results providers and aggregating results later either in different spots on results page or in the same list. This is done by abstracting each search results provider by means of a Location class. Important classes in federation object model are shown on the diagram below.


 image


As you can see, CoreResultsWebPart is connected to the federation OM through CoreResultsDatasource and CoreResultsDatasourceView types with the latter actually doing all the hard work interacting with the model. And also our objective, the EnableFQL property, exists in the FASTSearchRuntime class, which in turn sets this property on the KeywordQuery class, and as I showed at the beginning, this is what’s required to get FQL query syntax accepted.
As Todd Carter and the authors of Professional SharePoint 2010 Development point out, we need to extend the CoreResultsDatasourceView class in order to be able to control how our query is handled, and wire up our own class by also extending CoreResultsDatasource class. CoreResultsDatasourceView class creates a LocationList with a single Location object and correctly determines which concrete implementation type of ILocationRuntime to wire up based on search service application configuration. In other words, federation by default is not happening for the CoreResultsWebPart. There is another web part, FederatedResultsWebPart, and another view class, FederatedResultsDatasourceView, whose purpose is to do exactly that. With that, let us get back to our objective.
If we were using SharePoint Enterprise Search then we would be almost done, because public virtual method AddSortOrder(SharePointSearchRuntime runtime) defined in the SearchResultsBaseDatasourceView class would let us get our hands on the instance of ILocationRuntime. But since we deal with FASTSearchRuntime we are sort of out of luck. Yes there exists a method overload AddSortOrder(FASTSearchRuntime runtime) but it is internal! This is where the controversy that I have mentioned at the beginning comes to play: I was not able to find a better way than to invoke an internal member via Reflection. My way works for me, but keep in mind that usually methods are made private or internal for a reason. I used Reflection to access internal property LocationRuntime of the Location object. I don’t know why this property is internal. If someone knows or has a better way to get at the FASTSearchRuntime instance or the KeywordQuery instance – please leave a comment! Here is a code fragment showing extension of the CoreResultsDatasourceView and getting an instance of FASTSearchRuntime from there.

class CoreFqlResultsDataSourceView : CoreResultsDatasourceView
{
public CoreFqlResultsDataSourceView(SearchResultsBaseDatasource dataSourceOwner, string viewName)
: base(dataSourceOwner, viewName)
{
CoreFqlResultsDataSource fqlDataSourceOwner = base.DataSourceOwner as CoreFqlResultsDataSource;

if (fqlDataSourceOwner == null)
{
throw new ArgumentOutOfRangeException();
}
}

public override void SetPropertiesOnQdra()
{
base.SetPropertiesOnQdra();
// At this point the query has not yet been dispatched to a search
// location and we can set properties on that location, which will
// let it understand the FQL syntax.
UpdateFastSearchLocation();
}

private void UpdateFastSearchLocation()
{
if (base.LocationList == null || 0 == base.LocationList.Count)
{
return;
}

foreach (Location location in base.LocationList)
{
// We examine the contents of an internal
// location.LocationRuntime property using Reflection. This is
// the key step, which is also controversial since there is
// probably a reason for not exposing the runtime publically.
Type locationType = location.GetType();
PropertyInfo info = locationType.GetProperty(
"LocationRuntime",
BindingFlags.NonPublic | BindingFlags.Instance,
null,
typeof(ILocationRuntime),
new Type[0],
null);
object value = info.GetValue(location, null);
FASTSearchRuntime runtime = value as FASTSearchRuntime;

if (null != runtime)
{
// This is a FAST Search runtime. We can now enable FQL.
runtime.EnableFQL = true;
break;
}
}
}
}

By the way, another limitation of my approach is that using Reflection requires full trust CAS policy level. That said, finally we have arrived at our objective – we can set the flag on the FASTSearchRuntime, and it will understand our FQL queries. Our extended search results web part will show results as directed by the query (in the attached source code it uses XRANK) and leverage presentation richness of the CoreResultsWebPart.


Friday, January 21, 2011

Decks and Source Code from January 19th TSPUG Meeting

Thanks to all who came to Wednesday’s TSPUG meet up. I’ve uploaded a package with presentation and source code files. I was able to resolve the first of the questions unanswered during my previous talk about SharePoint search technologies. Although I meant to show it, I didn’t get to talk much about promoting user profile properties for selection in FAST user context for visual best bets. Basically as Steve Peschka has pointed out,  the first thing that needs to be done is permissions granted to the profile store. This will let you see the list of properties in his Property Explorer tool. Secondly, to promote profile properties to be available for selection in FAST Search user context administration section you need to edit value of FASTSearchContextProperties property of FAST Query search service application (SSA). I wrote a command-line utility for this purpose, which can be invoked as follows:

FASTProfilePropertyUpdater.exe –ssaId <FAST query SSA Guid> -action Add|Remove -property <User Profile property name>

Code fragment below demonstrates how SSA reference is acquired and the “Add” operation is accomplished. Full source code is a part of the package. Ideally you want to use PowerShell script to manage this, I just wrote an executable as it was easier to get it working quickly for me.

SPFarm farm = SPFarm.Local;
var settingsService = SPFarm.Local.Services.GetValue<SearchQueryAndSiteSettingsService>();
var serviceApp = settingsService.Applications[ssaId];
var searchApp = serviceApp as SearchServiceApplication;

if (null == searchApp)
{
Console.WriteLine(
"Cannot find search service application with the ID '{0}'.",
ssaId);
return 2;
}

string properties = (String)searchApp.Properties[FASTSearchContextProperties];
Console.WriteLine(
"Updating service application properties for property key '{0}'... Value before update was: '{1}'",
FASTSearchContextProperties,
properties);
List<string> propertiesList = properties.Split(',').
Select(p => p.Trim()).ToList();

if (ContextPropertyActions.Add == action &&
properties.IndexOf(propertyName, StringComparison.Ordinal) < 0)
{
propertiesList.Add(propertyName);
properties = string.Join(",", propertiesList.ToArray());
searchApp.Properties[FASTSearchContextProperties] = properties;
searchApp.Update(true);
Console.WriteLine("Property added. Updated value is: '{0}'.", properties);
return 0;
}

Also in the package is the source code for a web part demonstrating how to dynamically boost search relevance ranking from within an FQL query by using XRANK keyword.The demo of this web part didn’t go well, but the code is actually working (I dragged the wrong web part to the page during presentation!). Also someone has asked me about proximity-based filtering of results in FAST query language. Yes this is possible  - there is a number of keywords that support this.

Saturday, January 8, 2011

Automate Internet Proxy Management with PowerShell

The solution is very simple, yet the issue was so annoying to me that I thought to share this. I have a laptop and do work on it on site at my client and when I am at home. When I am in the office connected to my client’s network I need to use a proxy to access the Internet, while usually I do not need a proxy when I am located elsewhere. You normally set up a proxy using browser settings. With Internet Explorer 8 you open a browser, then go to Tools >> Internet Options >> Connections >> LAN Settings. There you configure the proxy parameters. As a part of configuration you can turn the proxy on or off by setting or clearing a Use a proxy server for your LAN check box, and as you switch the proxy the rest of your proxy configuration settings such as exceptions is preserved for you. Great!

My problem was that on almost daily basis I had to open the browser, navigate to settings then check or uncheck that box. I would come to work and forget to go through the steps so my browser would get stuck, and then I would go “oh yeah, I turned the proxy off last night”. Of course the reverse would happen at home… I tolerated this too long because I didn’t expect that a simple solution exists, but it does and here it is:

 HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyEnable key stores a DWORD value, either 1 or 0 to indicate if the proxy is enabled.

So I wrote two 1-line PowerShell scripts and added two shortcuts to them to my Windows taskbar: Enable Proxy, Disable Proxy! Here is how the script to enable proxy looks like:

Set-ItemProperty "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name ProxyEnable -Value 1

Then I’ve got excited, and wrote yet another rock star PowerShell script. Here it is:

$matches = @(ipconfig /all | findstr "myclientdomain")
if($matches.Count -gt 0){
    ./EnableProxy.ps1
}else{
    ./DisableProxy.ps1
}

That’s right I am using output of the ipconfig command to determine if I am on the client’s network and if yes then I turn on my proxy. After I have added a scheduled task running at user logon to invoke this script, the quality of my professional life has improved.