FAST Search for a specific External Content Type

I hadn’t really had a chance to do much with FAST Search, beyond your standard demo-ware installs and use cases, but I’ve been anxious to have an excuse to dig into it more.

The combination of beefed up BCS/BDC support in SharePoint 2010 plus FAST search seems like it would be a pretty powerful combination, especially if you have fairly large sets of data in your LOB applications and you don’t want to hammer them with constant, open-ended queries.  In this case I was looking to be able to limit a search to return results of only a specific External Content Type (ECT).  In the past, with SharePoint Search I’ve been able to just use the ContentType managed property for this (there’s a default mapping that brings in the ows_ContentType crawled property and maps it to this).  However, it seems that for external content types, this property does not get set.

One option here would be to just setup a unique content source for each type of BCS entity that you want to bring in, and use the content source as part of the query.  That seems pretty ridiculous for scenarios where you either have: a) a lot of entities/ECTs or entities that are related to one another.  Fortunately, poking through the available crawled properties in my FAST Query SSA I found that there is one called “entityname” under the “Business Data” category.   Mapping this to the ContentType managed property worked like a charm.  I can now go ahead and send a query like the one included below to the SharePoint Search Web service and get back only results that are of the type that I care about. BTW, many thanks to the author of the FAST for SharePoint Query Tool. It’s the FAST-specific version of the SharePoint Search Tool I mentioned really liking in a previous post.

<QueryPacket Revision="1000">
  <Query>
    <Context>
      <QueryText language="en-US" type="FQL">filter(contenttype:equals("MyBCSEntity"))</QueryText>
    </Context>
    <SupportedFormats Format="urn:Microsoft.Search.Response.Document.Document" />
    <ResultProvider>FASTSearch</ResultProvider>
    <Range>
      <StartAt>1</StartAt>
      <Count>20</Count>
    </Range>
    <EnableStemming>true</EnableStemming>
    <IncludeRelevantResults>true</IncludeRelevantResults>
    <ImplicitAndBehavior>true</ImplicitAndBehavior>
    <TrimDuplicates>true</TrimDuplicates>
    <Properties>
      <Property name="Rank" />
      <Property name="Title" />
      <Property name="Author" />
      <Property name="Size" />
      <Property name="Path" />
    </Properties>
  </Query>
</QueryPacket>

Download SharePoint User Profile Photos Using PowerShell and Search Web Services

Have you ever wanted to grab all of the user profile photos in your SharePoint environment, whether it was to make sure none of them were inappropriate or just to have a laugh at some of the creativity?  It can be pretty entertaining, even if you don’t have an HR-ish reason for wanting to check on them.

I was recently asked to put together some way for a small group of users to easily look through all the pictures that had been uploaded.   The idea was for this to work from a client machine with no server-side installs would be needed.  This meantusing the object model was out (no Client OM in MOSS remember).  Fortunately in this environment all the user profiles are indexed to enable the “People search” capability, so the search web service is a handy and performant way to grab a few profile attributes for all the users who have submitted a profile.

I started by figuring out the syntax of the query I wanted to use to get the results back.  If you don’t already use it, I can’t recommend the SharePoint Search Service Tool enough.  The standalone executable is a free download from Codeplex that some really sharp people from Microsoft were kind enough to share, and it’s a very educational and useful tool if you want to dig into the SharePoint search engines native language a bit.  Generally I find I can just pick the scope and first few attributes I want to include in the results and then tweak the query syntax by hand.  One of the most powerful aspects is that the search tool lets you use SQL Syntax for the queries, which opens up a whole bunch of possibilities that the OOTB search web parts can’t do.  In the case, the crux of the query was to use the “People” scope, pick the attributes I wanted”, and make sure I only grabbed profiles that actually had a PictureURL populated.  You might think you’d be able to do something like PictureURL is not null, but despite being legitimate syntax it doesn’t appear to actually work.  For my purposes, I just ended up a “like (h%)”, which got the job down even though it may not be the most efficient predicate ever.

At this point, I had a nice chunk of XML for the QueryPacket to send to the search web service, as well as a pretty good idea of what the XML in the response was going to look like.  I don’t claim to be an XSLT master, but it’s not rocket science to turn the XML response into a really simple HTML page that shows the thumbnail versions of all the images, especially if you have a nice XML/XSL editor at your disposal to help with debugging.  Using the auto-generated thumbnails is highly recommended, by the way, unless you like seeing your browser sweat bullets trying to layout thousands of 100K-5MB images on a single page.  Like I mentioned before, it’s pretty funny what people will upload even though it all ends up in a 150×150 square.

I now had a one-off version of the desired result, but didn’t want to want to burden the actual customers with needing to know how to use things like the Search Service Tool, even as much as I like it.  I needed something quick and easy to automate this process, and for most SharePoint folks these days “quick and easy automation = PowerShell”.  After saving the QueryPacket XML and .xsl file for transforming the response, I just need a handy way to call the search web service from PowerShell.  At first I used the “old school” method of just generating and compiling a proxy stub with wsdl.exe and calling that from PowerShell.  After a bit more search engine fun, I found that there is a New-WebServiceProxy cmdlet built into Powershell to handle this for you.

Now I have 3 easy-to-distribute text files, and as long as the user can run the PowerShell installer, which is pretty straightforward, they can just unzip the files, right-click the .ps1 to choose “Run with PowerShell”, and enjoy perusing a large HTML file with a bunch of images that link back to the user profile page.  The actual HTML could be made much fancier, and obviously you may care about a different set of profile properties than I’m pulling.

This was a fun little exercise that actually served a practical purpose in this case, but might otherwise be a nice, fun page to show a client for whom you’ve deployed SharePoint My Sites.  You can download and play with the source files however you wish.  I’d love to hear feedback, or other ways to extend this idea in the comments.

Off to a new frontier

Well, I’m going to be moving out of a fairly length tenure at Amgen to a consulting gig at Slalom Consulting.  I’m really excited about it, and one of the reasons is that I think I’ll have more leeway to blog about some of the interesting tech stuff we’re working on or that I’m personally looking into or dealing with.  Most of it should be right up the “MS Developer” alley.

The company seems to have a really amazing roster of consultants all over the place, and is truly kicking some tail in the Microsoft tech space.  It should be a blast.

SharePoint 2010 Class Notes – Day 3

Large list throttling is definitely a good thing, but will take some getting used to.  Steve Peschka has a nice writeup on some of the implications of list throttling.

Ted gave one of the better explanations of LINQ I’ve heard or seen, which I didn’t expect to get in a SharePoint class necessarily.  It’s welcome though because the LINQ support in SP2010 is pretty handy.

MSDN has an article on how to see what CAML gets generated by the LINQ to SharePoint provider.

SharePoint 2010 Class Notes – Day 2

We’re talking about sandbox solutions in SharePoint 2010 today.  This seems like it will be a very popular capability amongst the power user audience, but could be tedious to manage a bunch of site collections and use the same set of sandbox solutions.

An interesting solution, um, solution might be to have to establish a central repository of sandbox solution WSP files, then have some kind of Silverlight app that would let you register a series of site collections you manage, and then deploy a set of selected solutions to all of them.   Think of it as a way to manage “my” slice of the farm, without using farm-wide solutions.

Solution validators are an interesting concept.  It seems like two very useful validators would be a simple whitelist or a simple blacklist, perhaps that just read a list of solution identifiers from a SharePoint list or simple database table (if you want to avoid a circular dependency) somewhere.

Looking at visual upgrade, there are some weird issues if you want to keep running with the old UI/master pages.  They’ve stapled a new feature called Wiki Page Home Page to the team site definition to use the new Wiki Page library, which sounds like a cool idea on the surface.  However, if you have a site that uses the old UI, the new wiki pages are pretty much totally unusable because the old editor doesn’t know what to do with all the new rich editor content.  It appears though, that if you just delete the “Site Pages” library on a team site, it will revert back to the old behavior and just use the default.aspx page that has plain old web-part zones on it.   It would probably be cleaner to just deactivate the feature though.

Some VS2010 “magics”:

• prop[tab][tab] – Create a new property
• object.Event+=[tab][tab] – Stub out an event handler
• override [methodname]

RPC – Form Submission Cannot Be Processed Error

We recently had a fun little issue where a seemingly random set of users with varying combinations of Windows 2000, XP, and Vista as well as Office 2003 and Office 2007 started having issues editing documents from SharePoint documents libraries.

This didn’t effect all our users, which made it trickier to debug, but when we were finally able to do a Fiddler trace it appeared that the calls failing were all going to the /_vti_bin/author.dll library.  This is a long-standing holdover from the bad-old days of FrontPage and FrontPage extensions that still gets used in several scenarios by Office clients (including SharePoint Designer).

The RPC error we kept getting was “Form submission could not be processed because it exceeded the maximum length”.  This seemed really odd since the actual request size in these traces was only a few bytes.  We knew our Maximum Upload Size value was bigger than a few bytes, having just gone through an exercise to increase it to 200MB.

I’ll just cut to the chase rather than bore you with all the dead ends we went down (Google was unfortunately not a lot of help on this one).  It turns out someone had changed our Maximum Upload Size to 2048MB.  I don’t know why this happened, but it’s slightly bigger than the maximum size noted in the stsadm docs for the max-file-post-size (2047MB max).  If you spend a lot of time with powers of 2, you’ll probably notice this is suspiciously close to one of the magic numbers that tends to result in things like overflow errors (i.e. it’s right at the maximum size of a signed 32bit integer).  My guess is that the old FrontPage RPC stuff still uses 32-bit integers to store this kind of thing, and it probably “rolled over” to look like a negative number.  So any calls to author.dll were “too large” no matter how big they actually were.  I don’t know for sure, but it seems like a high probability.

Anyhow, just change the Max Upload Size back to a more reasonable value resolved the issue and our Office clients  were happily RPC-ing away again.

Highlights from SharePoint 2010 Hands-on

I thought I’d do a quick post to capture some of my favorite (and not-so-favorite) findings from spending some time with SharePoint 2010 in the hands-on-lab room.  I’ll update this as I get time.  Perhaps it will even turn into a nice “Top 10” list or something.

Highlights

1. You can now add field/column validation right from the UI using the same Excel-style formulas previously used for filters.  There are limits (i.e. you only get one  “error message” for the whole form instead of for each column), but this will be sufficient for many of the simple cases
2. You can seamlessly replace the edit/new form for a list with an InfoPath version by launch InfoPath from right in the ribbon for the list.  It’s really slick and will work around a lot of those pesky “Why can’t I lay my columns out this way and label/describe them better” requests.
3. The new list view web parts are *crazy* flexible and nice and AJAX-y.  You can also embed them all over the place (e.g. inside of a wiki content section – more on this later)
4. The rich text editor combined with the ribbon UI is absolutely brilliant.  This blows away the editing interface for any other similar tool I’ve seen, and the promise is that it will work fine across browsers too.  Haven’t had a chance to test that, but I doubt if they would claim it if there wasn’t a lot of work going into it.  Our users are going to love, love, love this editor.
5. Tagging is *everywhere* and extremely powerful.  This feels really well thought through, and in retrospect I’m glad we avoided trying to graft on a third party product.
6. Document IDs – yup, you can finally have unique document ID that will never change and link to it from anywhere.

Lowlights

1. List to Excel integration is still via the same crufty .iqy file mechanism that has been around for several SharePoint iterations now.  Not only will this mean years of the annoying/scary error dialog when opening a list in Excel, but it likely means no way to have a published Excel sheet (in Excel Services) that points back to list data.  C’mon Microsoft, is it sooooo hard to make Excel use the same integration mechanism that Access does (i.e. web services)

Jeff Teper Keynote and live demos – SPC 09

Jeff is a corporate VP in charge of SharePoint who has apparently been around it for a very long time.  I don’t really remember him from previous events for SharePoint, but I’ll take his word for it.

He put up a Bill Gates “1 slide vision” document from 1999, and it’s pretty funny how well it still applies to the challenges around integrating all the stuff SharePoint deals with together.  The man definitely had a vision.

Ugh, so far this talk is like the overly-long Academy Awards thank you list.  I’m sure everyone feels much better about themselves.

We’re going through the new version of the SharePoint “cheese wheel” or pie.  It’s now all blue, and has some tweaked titles, but largely still very similar.  I’m sure will be seeing lots of this.

Communities is one of the new concepts in the wheel.  It’s basically lumping together some of the collaboration stuff they already had (blogs, wikis, team sites, etc) with more of the social networking capabilities (tagging, “folksonomies”, meta-data management, expert identification).  The My Site experience and the profile has been revamped considerably, including a Silverlight org-chart that may be the Outlook Address Book killer.  We’ll be needing to take a close look at this at work.

ECM Scenarios.   They are taking the scalability to whole new level.  Millions of documents in a folder or list, 10’s of millions in a library, federated scenarios to make a seamless experience of 100s of millions across several libraries.  This will definitely require some in-depth investigation later, but it certainly sounds promising.  They are going for the jugular vs Documentum/Livelink here.  Second big applause item here – content types can be managed at any level, including across farms.  They can store large objects like videos for digital asset management scenarios (goodbye Podcast Kit for SharePoint).  “Document sets” are a new feature that is very much like virtual documents in Documentum.  They’ve ramped up the governance, records management, compliance facilities.

Search is changed pretty significantly as you can now have either “standard” search or FAST search plugged in.  These are supposed to have a consistent programming model, but he did not say if it is the same as the existing model.  My guess is no.  They are trying to leverage the fact that they know our organizational structure to increase search relevancy.  I’ve been hoping for this for years, as they have significant competitive advantage there vs other search platforms given the OS/Office footprint.  Also doing things like automatically figuring out name “typos” in searching for people based on the way the names might be mispronounced phonetically.

On to BI.  They are pushing the whole PerformancePoint suite into SharePoint as a throw-in now.  Excel services now looks like a much more complete product.  Things like charting web parts, sparklines, etc are now built-in.  On a very geeky note, Excel calculation services are now available as REST feeds, so you can link to specific Excel data from anywhere on the web.  Sweet!  They’re going to have some ways to bind  Visio charts to data and then expose this in SharePoint.  That could either be useless or tremendous, depending on the implementation.

They are finally unveiling “Project Gemni” as SQL Server PowerPivot.  This is something that can either be run on the client (in an Excel flavor) or the server (a SharePoint flavor) to slice and dice millions of rows very rapidly.

Admin Notes

Big picture, their SharePoint Online offering made them highly motivated to have a scalable way to manage large SharePoint environments, so there have been a lot of improvements here.  They finally put the Admin Web UI through their usability labs.  If you’ve ever seen the Central Admin page in WSS 3, you’d know this was a long overdue idea.

PowerShell is fully baked into this release for the command-line administration of SharePoint.  I don’t think this means the death of stsadm.  It’s more of a complement.  Also you can manage the farm remotely from a client machine (Windows 7).  Yes!

Centralized logging database for SharePoint environment that will have a publicized structure so we can use whatever we like to report against it.

Upgrade experience should be much more seamless than 2003-2007.  As they already demoed in preview videos,

Demos

Someone named Jared up to do demos.  No, not the Subway guy.

He’s showing the ribbon UI and the rich text editing.  Wow, he just popped into “wiki syntax” to point to an arbitrary piece of content in the site.  It’s hard to describe but, typing the [[ brought up a way to pick any other available pages, kind of like Intellisense in Visual Studio.  It’s really slick!

Showing the new ribbon UI also includes the ability to perform operations like “Checkin” on multiple documents at once.  This is one of those things that seems like it always should have worked that way, but it finally does now.

Very slick integration between the Office Clients and SharePoint.  For instance, now opening a SharePoint site in the File dialog in office lets you browse by meta-data not just treat it like a bunch of folders.

You can definitely tell they’ve listened to customer feedback on pain points.  That has always been a Microsoft strengths.

On to the new MySite/social stuff.

Hmm, a tabbed interface for all the sections on your profile  page (e.g. organization, my blog posts, etc).  I think I knew someone once who liked that idea and proposed it as a customization for SharePoint 2007 My Site profiles.   There’s now a “work notes” section on profile pages that is a blatant rip-off of the Facebook Wall, but lots of people seem to like that.

Ratings are built-in and ubiquitous.  Nice!

They are so going to kill Google Docs and Spreadsheets with the online versions of the Office apps.  In demoing the “co-editing” features, Jared has a great line.  “The nice thing about co-editing is that it allows me to watch other people do my work for me.”   Well, when you put it that way, sign me up.  REST-ful feeds for Excel look pretty awesome too.  He just showed how to link to a live chart directly from Excel services using the “Insert Picture” capability in  a blog post.  It’s a really powerful concept.

He’s really flying through stuff here.  Showing search, which now has wildcards built-in.  People search is rerally slick with that new Silverlight Orgchart gizmo (that’s not really a new technical term.  Why not, we use gadget and widget already?).

So PowerPivot appears to scale pretty well.  He’s working on an Excel spreadsheet reporting 101 million rows.  It’s smoking fast filtering data.  Not sure what this is based on in the backend.  It might be use Analysis Services cubes or something.  Combining this with the ability to publish a workbook to a SharePoint site is extremely powerful.  Even with an investment in another enterprise BI tool (e.g. Cognos,  Business Objects) this is something that bears looking into.

It looks like there is some flavor of SharePoint Workspace/Groove available for Windows Mobile, but no demo so it may not be baked yet.

Arpan Shah handles the admin demos.  There is a new “Health Analyzer” section in Central Admin that will try to identify issues in your farm based on a rule set.  This should be pretty helpful, since there are so many places to look for this stuff now.

He’s also showing the ability to set thresholds on large lists to allow lots of items in a list, but put governors in place to keep inefficient use of them from overwhelming your environment.  Not that this could ever happen in SharePoint 2007 or anything.

SPC 09 Keynote

SharePoint Conference is huge this year.  There are about 7400 people, so it’s almost twice as big as last year.  The keynote venue is gigantic.  Technically seems like a very well setup operation.  The web site is really nice for managing all the session, then being able  to access them via a phone.  All tech conferences should be doing this at this point.  No excuse vendors.

Tom Rizzo, whose a Director or something or other SharePoint related at Microsoft (just kidding Tom) is M-Cing the keynote, but people are definitely here for the demos and Ballmer.  Wow, allegedly two couples are getting married at the conference.  That’s, um, some kind of special.

They are rolling a video clip of high profile SharePoint sites.  There are some *big* (think Fortune 100) names in there.  I don’t think we need to worry about whether we’re going out on a limb to recommend SharePoint for ssites these days, whether they are Internet facing or internal.

And, hey, it’s Steve!  Not as cool as turtleneck Steve, but style points  don’t win the enterprise.  Hey, apparently this is the  first speech in a while he doesn’t feel obligated to start with a commentary on the economy.  I guess he feels like the SharePoint economy is doing just fine thank you.  If you look at their sales numbers, you can see why.  He mentions that they have a video camera at the back of the audience where they can tell how many people have some kind of laptop, tablet, etc fired up.  It turns out not very many.   That’s one of the points he’s trying to make is that we still have work to do on device form factor (see their tablet PC initiatives, Surface, etc).

First big datapoint – public beta in November – for SharePoint 2010 and the rest of the Office 2010 suite.  Other than that he hasn’t said anything that wasn’t already on the MS SharePoint website.

Big trend stuff:

Move to cloud hosting (SharePoint Online)
Push to external facing sites
Emphasis on composite and “mashup” apps.  Of course, this was Plumtree’s vision circa 2002.  Turns out this is harder than you’d think.

Wow, Steve’s actually a SP user himself.   I guess this shouldn’t be surprising, but it still kind of is.  He just talked about how he setup a collaboration site for all of his senior leadership team with their latest sales numbers, different documents, etc.  It didn’t sound that complicated, but, hey, he’s the CEO and he’s an actual hands-on user.  I’m sure most companies would kill for that kind of adoption.

Based on his comments, it’s clear they received (and responded to) an absolute ton of feedback about the developer experience for SharePoint and it’s lack of “1st class citizen” status in Visual Studio.  This is good.

SharePoint Online is being taken very seriously.  Over 1,000,000 users.  Glaxo is moving 100,000+ users to it.  Pretty much the whole set of end-user features is available in the cloud version, and some of the developer features are available (see the “Sandboxes” stuff in the demo section).  They are more than happy to support “mix and match” capability between hosted and on-premise environments.  Of course, this is probably also the ideal setup for them from a licensing standpoint.  You’re paying for stuff twice.

Microsoft *really* wants us to be able to build Internet-facing sites with SharePoint 2010.  Some interesting examples.  Kraft Foods has consolidated 200 web sites into one SharePoint infrastructure.  100M page views/month.  Marketing people are able to publish new content entirely without  IT involvement.  They actually have a new SKU they are launching to support this “SharePoint Server for Internet Sites”.

Ugh, a big slide of licensing soup.  I just don’t have the energy right now, but it looks like generally we’re fine internally and may be better off for external-facing stuff.  Also, apparently “SharePoint Foundation” is the new name for WSS.  I’m not sure if this helps or hurts in terms of the old confusion between WSS/MOSS.

Interestingly, Steve says that SharePoint today probably does not look that much like what their roadmap said 5 years ago.  They have really been pushed by their customers in directions we want.  That’s a good thing, and they seem happy to go with it.

Demos 

First big applause point of the day.  He’s running the demo on Windows 7!  Yay, no more having to setup server operating systems to do SharePoint development, or at least basic development.

SharePoint Designer now actually looks like an Office app, ribbon and all.  So he’s showing a concept called “External Content Type” which basically allows external databases/web services to be mapped into the SharePoint/Office universe.  This is kind of just the Business Data Catalog and the old Data Sources concept in SharePoint Designer.  It’s not clear from this demo how well this will work in the real world, but it certainly has potential.

Now he’s showing the integration with Visual Studio 2010.  It’s definitely.  There’s a visual web part designer, which should be pretty nice as a productivity enhancer for developers.  Much nicer solution building capability now.  Also, it looks like their using the “mapped” directory capability SP already has to let you add supporting resources like images and styles to your.

There’s a “developer dashboard” console that can be brought up on a SharePoint page now.  It has all kinds of super useful debugging information.  This will be *huge*.  Congratulations Microsoft, you caught up to the version of Plumtree they had 4 years ago, well almost.

Sandboxes are a new capability to allow certain groups of users more rights than they otherwise might have to deploy web parts and things without having full admin rights.  You can also monitor these for bad behavior and shut them down.  I suspect Microsoft was their own major customer for this feature because it will allow them to host SharePoint as a cloud-based service much more easily.

Web Content Management has been improved quite a bit.  Page editing now uses the ribbon UI.  They *finally* have a way to support XHTML (i.e. compliant) web pages.  Can change page layout on the fly (similar to picking slide layouts in PowerPoint).  The rich-text editing is now cross-browse and (allegedly) generates much cleaner HTML.  There is a built-in media player web part now.  Couple niceties – automatic spell checking for publishing pages.

They demoed a very click Silverlight web part to “search” a product catalog using the FAST search capabilities without typing any keywords.  It’s unclear how much development this took though.

QA

Q: Will their be a “Windows Update-like” page that has all the updates, service packs, etc?
A: Steve says “the right answer is ‘yes’”, but not sure of the actual answer.  Tom is clearly a bit embarrassed, but says they are working hard to make this better

Q: How will SP keep up with “web-speed” development cycles?
A: Steve basically thinks it’s a myth that development happens more rapidly just because it’s on the web.   He’s probably right

Q: What about social computing in SP (nothing like a softball question)?
A: Lots of social networking stuff in SP.  Steve takes the opportunity to plug FAST “semantic” search

Q: What are they doing to make governance better?
A: Tom says governance is mostly a process issue.  Steve shows why he is CEO and makes the cogent point that one of the most important things SharePoint Online will do is allow them to “industrialize” the processes  around supporting large SharePoint environments and then share it with everyone else.  Great point.

Q: How about Mac support?
A: The browser-based pieces should work across browser, including Safari.  There will always be things that work better

Q: Can  we use the new developer tools with the “old” SharePoint 2007?
A: Nope, but maybe in the future

Q: Teed up question from Tom to Steve (but very relevant) How willing are enterprises to embrace social computing?
A: Steve talks about a CEO of a Fortune 50 company that he went to B-school with whose CIO says “there will be no facebook in our enterprise”.  He basically says of course you don’t want your corporate data going on the actual Facebook and Twitter, but you should be able to feel comfortable putting those types of data on your own SharePoint environment.  They are trying to give us the ammo to make this case at the conference.  Let’s hope they’re successful.

Tap, tap is this thing on?

Just checking that my Windows Live Writer setup is working from the EEE PC.  I’m anticipating a blog-alanche from the SharePoint Conference 2009 starting tomorrow, so I want to be sure I’m good to go.

Let the games begin.