Countdown to launch: Importing data

In the eleventh in his series on implementing SharePoint 2010, Robert Schifreen tackles security and edges towards live data.

Securing a complex system like SharePoint is itself complex. The web forums are full of postings that state categorically that you need an application firewall in the form of Microsoft's Threat Management Gateway (TMG) between your SharePoint farm and the internet.

Possibly coincidentally, many of those postings come from Microsoft. Assuming you have a corporate firewall, and you use the Windows firewall on your servers, you may well decide that this is sufficient. It's something we discussed over the course of many weeks, and opinions were definitely divided.

Also, while it's true that SharePoint 2007 required TMG in order to use forms-based authentication, 2010 does not.

Microsoft recommends that, in addition to a standard antivirus product on the SharePoint and SQL servers, to protect at the server level, you also use a separate SharePoint-aware antivirus product on the SharePoint boxes too.

This scans incoming documents and ensures that no infected files end up in your content databases. But are you really bothered about infected files ending up in your SQL content databases? They can do no harm in there, and the infection will be detected as soon as the file is accessed by a user. Is it worth the overhead of having another CPU-intensive process running on the servers?

Design and branding

A SharePoint site, especially one that isn't a public-facing website, tends to look like every other SharePoint site on the planet.

You can, if you wish, customise it in any way you want. At the simple level you can replace the SharePoint logo with your own. If you want to get clever, you can replace the default html and css template files and change the complete look and feel.

A SharePoint site tends to look like every other SharePoint site on the planet.

However, major branding exercises can take time, can lead to some things breaking for inexplicable reasons, and means that your users won't be able to easily find their way round the numerous books and websites dedicated to helping people use SharePoint.

Our current plan is to brand our SharePoint intranet so that it looks and feels vaguely similar to our existing public website. We're currently at the Photoshop mockups stage with our design agency. If you don't have the time or money to do that, there are lots of cheap and free SharePoint designs around, as well as some generic minimalist templates from which you can start creating your own look and feel. Search online for "starter master pages" or Heather Solomon.

Importing data

Before we can go live, we'll need to import our users' document files from the existing network shares into SharePoint.

There are products that remove the need to do this, and which can make documents in existing file shares appear to be in SharePoint databases. But that sounds like a nightmare to administer and I'd much rather know that a full backup of all our SQL databases is guaranteed to contain every user's files.

The data that we need to import falls into two categories. Firstly there are a few hundred thousand files in our users' personal network drives.

We can either import everything in bulk, via an automated tool, or we can give users a couple of months to move everything they want to keep, after which we'll turn off the old shares or make them read-only. Either way is a relatively easy task.

There's a good bulk upload tool, which improves on the facilities in out-of-the-box SharePoint, available for a couple of hundred dollars, which will make it easy for users to copy stuff across. Or they can just use WebDav from Windows. Or I could probably do the whole thing in PowerShell, because there's enough information in our Active Directory for me to find their current files, check that they are staff rather than a student, and copy the files to the correct SharePoint library.

If we automate things from our end, we will need to pre-build all of the staff My Sites before we go live.

This is not the way that SharePoint works by default, which is that a user's My Site only gets created when they log into SharePoint for the first time. This won't work for us, because if we need to automatically copy a user's files from their network share into SharePoint, their SharePoint site needs to exist. I managed to come up with some PowerShell that iterates through the SharePoint user profile database and creates a MySite for all staff automatically.

Preserving permissions

The second category of data that we need to import is the content of our departmental (rather than personal) shares. There are around 60 of these. In theory, this should be a very easy job too. We set up 60 document libraries and I drag 60 directory trees from the shares to SharePoint.

There is a dire shortage of affordable migration tools that can preserve file permissions.

But there's a major obstacle in the way. We've always allowed departments to manage the security of the files and folders in their shares, so they can grant permissions to specific users or groups. There are currently tens of thousands of files that have non-standard permissions, and we are very keen to preserve these.

There is a dire shortage of affordable migration tools that can preserve file permissions. It's not an easy thing to do, because the permissions model in SharePoint is completely separate from that of NTFS, and the permissions available are different too.

The market-leading products in this area, some of which I've tested and which do appear to work just fine, cost in the region of £20,000. I'm reluctant to spend such a sum on a one-off task that will probably be accomplished in an afternoon once initial testing is completed.

There are plenty of migration tools in the sub-£500 category but few of them preserve permissions. One that does, from ShareGate, looks promising and I intend to test it as soon as I can. There's also a free tool on sourceforge, called File Migrate Console, that claims to do it but which I can't get to work at all. I might just have to dig out Visual Studio and my copy of "C# For The Downright Stupid".

On the subject of expensive migration tools, special mention should be made of one particular well-known purveyor of such. When I called to speak to their pre-sales department, and explained that we had a few tens of thousands of files to move, his first question was regarding how many files I'd be prepared to lose. His second was to ascertain that we'd definitely be taking a backup before we started the migration process. I didn't pursue them further. I reckon that's £14,000 well saved.

Next: In the final installment: All over bar the users...

Robert Schifreen has reported on and implemented online technology since the early 1980s. His latest project has been a large SharePoint 2010 installation in tertiary education. We will be serialising his experiences, positive and negative, in getting it to the stage where it's ready for action; the entire series will also be available as a downloadable white paper.