File Migrations with SuiteMoves

Partners can use the SuiteMoves tool to migrate a file system to Office 365.

How it works

SuiteMoves is designed to quickly migrate a file system from a Windows Explorer environment to Office 365. It copies the exactly folder and file structure as it appears in Windows Explorer without you having to do anything specific first. During the migration the SuiteMoves tool will:

  • Handle all of SharePoint's restricted characters in folder and files names by simply stripping them out. The exception is the & character which SuiteMoves replaces with a +. A full list of restricted characters in SharePoint Online is available here.
  • Handle SharePoint's restriction that a file name be no longer than 128 characters by truncating all file names to a maximum length of 50 characters. This is not a configurable item in SuiteMoves, the maximum file name in SharePoint after a migration with our tool is 50 characters. Files with long names are cut at the 50 character point. If this creates duplicate file names the name is appended with a number.
  • Handle SharePoint's restriction that the full URL created for any item be no longer than 256 characters by truncating folder names. The original folder name is kept in a piece of meta data against the folder and this is displayed to users in both SuiteFiles web and Outlook add-in so they see the full names for the folders. This is not a configurable item in SuiteMoves.
  • Migrate all folders and files to the Shared Documents library in the SharePoint team site. The List View Threshold constraint of 5,000 items in a list is not relevant for SuiteFiles users as they do not use the standard SharePoint Online user interfaces. A document library in SharePoint Online can have up to 30,000,000 items stored in it. If the Shared Documents library doesn't exist the SuiteMoves tool will create it.
  • NOT copy any permissions from the existing file system.
  • NOT copy any file version history or meta data from the existing file system.
  • Set the last modified by field to the name of the user account running the migration.
  • Set the last modified date/time fields to the date/time that the file is uploaded into SharePoint Online.
  • NOT run "delta" migrations and pick up changed files after a first migration has been run.

SuiteMoves is designed so you don't need to spend time rearranging the file system or thinking too much about the process. The goal is to make it as easy as possible for you, the person running the migration.

What files are migrated?

All files are uploaded to the Office 365 SharePoint Online site except:

  • Any file listed on this Microsoft page,
  • EXE, DLL, thumbs.db, DS_Store, LNK, URL files,
  • Files in system folders,
  • Files with a name but no extension,
  • Files with an extension but no name, and
  • Files starting with . or ._ in the file name.

How long does it take?

The time taken to complete a migration depends entirely on:

  • The volume of data being uploaded,
  • The upload speed of the internet connection,
  • The processing power of the computer being used.

As a general rule we wouldn't run a migration where the upload speed of the internet connection is less than 5Mbps. This will be bound to fail. In this case we'd take a copy of the files to a location with a better internet connection. Ideally you'd want a connection of at least 10Mbps to run a migration. Use  www.speedtest.net to test the internet connection speed if in doubt. 

We like to run migrations over a weekend. With a good internet connection it's reasonable to get up to 100GB uploaded per day.

What it costs

We charge our partners a one off fee when you register a site for a migration. This registration is a license fee to use SuiteMoves for the specific SharePoint Online site being migrated to. At this point there is no time limit or volume limit on this license for the site. You need to register SuiteMoves for every site you wish to migrate to. We will invoice you for the license fee in the next partner invoice period after you register the site.

You can then charge your customer a price that you see fit for the migration. For your information, if we run a file migration ourselves we charge based on the volume of data migrated using the formula:

  • $750 for the first 20GB
  • $15 for each additional GB

You are welcome to use this formula or work out a different price based on the actual service you're providing and what best suits your customers.

Registering the migration

The first step in the migration process is to register the customer's SharePoint Online site. This is done in the SuiteFiles partner portal:

  • Log in to the SuiteFiles Partner Portal,
  • Click on the SuiteMoves menu option then Register a Migration 
  • Enter the URL of the site that you are migrating to, check the box acknowledging the migration license charge and click the Register Migration button. Please ensure that the URL is in the correct form and include the https:// prefix

NOTE Clicking the Register Migration button means you will be charged the fee for the migration at our next invoice period whether you have run the migration or not.

Setting up the migration computer

A computer, or server, is required to run the migration from. The prerequisites for this computer are:

  • 64bit Windows operating system running as a minimum either Windows 7 SP1 or Windows Server 2008 SP1 (note that there is a cut down version of Windows Server known as FE and this will not support a migration),
  • 8GB RAM,
  • It has internet access,
  • It has access to all of the files that need to be migrated,
  • The Microsoft .NET Framework version 4.6.2 is installed (note on some computers this may force a reboot),
  • The latest SharePoint Client Components are installed (note any computer that cannot install this component cannot be used to run a migration),
  • The SuiteMoves tool, and
  • The Microsoft Message Queue (MSMQ) Server feature turned on. This is done by going into the Control Panel, selecting Programs and Features then clicking Trun Windows features on or off Then search through the list of Windows features and check the option for Microsoft Message Queue (MSMQ) Server and click OK. This will install the MSMQ components and possible also require a computer restart once done. 

Preparing the files to migrate

While SuiteMoves is designed so that it can be sued without thinking about restructuring the file system before a migration there are some things that you may wish to consider:

  • Are you migrating files that aren't needed? Maybe old files or large backup files are included in the file system. If you don't need to migrate some files they need to be removed from the files being migrated.
  • Having all of the files in one place on the file server - they should all be migrated at one time as SuiteMoves doesn't support "delta" migratons. Users of the files should also not be working on the source documents at the time of the migration as changes made to them will not be migrated.
  • The folder structure being uploaded. The top level of folders to be migrated will be the top level folders in the document library after the migration is run.
  • Does the customer run a customer centric business? If so they should have all of their client folders immediately under a top level folder called Clients. This will allow them to take advantage of the client search features in SuiteFiles.
  • Is the customer an accounting practice using the Xero Practice Manager that wishes to integrate with SuiteFiles? If so then their client folders should be named to match the client records in Xero Practice Manager.

Preparing the SharePoint site for migration

Before starting the actual migration the SharePoint site also requires a little tweaking. To do this you require a local Global Administrator user account in the Office 365 tenant. This user account does not need to be licensed and it's the account that you should use for actually running the migration. To prepare the site you need to:

  • Log in to the SharePoint Admin Center using the global administrator credentials. The URL for the admin center is https://domain-admin.sharepoint.com where domain is the Office 365 tenant domain for the account.
  • For the site collection you are migrating to add your global administrator credentials to the Site Collection Administrators by selecting the site collection, clicking Owners and then Manage Administrators followed by OK,
  • Set the SharePoint Lists and Libraries experience to Classic experience by selecting settings then click OK at the bottom of the screen, 

Testing SuiteMoves

Before starting a migration you should always test that SuiteMoves is going to work properly before actually starting. To do this you need the latest SuiteMoves application saved to the migration computer plus a folder with a file or two in it.

  • Run the SuiteMoves.exe file
  • Log in using the URL of the site you're migrating to plus the admin credentials you've created for the Office 365 tenant, click the Login button 
  • Drag the test folder on to the SuiteMoves drop zone - this folder will become a top level folder in the SuiteFiles site 
  • SuiteMoves analyses the files selected for migration and what's already in the SharePoint site then provides feedback. Click the SharePoint button 
  • This starts the process of creating the folder structure in SuiteFiles. Once done the SuiteMoves app will tell you that it's finished plus you'll see a .migrationmodel file and finaloutput.csv file in the folder SuiteMoves is being run from. The folder structure is created in SuiteFiles 
  • The next process is to upload the files. Run the SuiteMovesFinalUploadsTool.exe, log in using the same credentials as above, check the Do Upload box (you may want to expand the size of the app on the screen at this point) and click the Choose Migration Model button 
  • Select the .migrationmodel file created and saved into the folder you're running SuiteMoves from, click the Open button 
  • Each file is then uploaded into SuiteFiles until the process is finished. Click the Export Current CSV... button to generate a full migration report. When the report is generated click OK 
  • Back in the folder SuiteMoves was run from there's now an output.csv file. This contains all information about the migration. Open the CSV file in Excel and add filters to the columns. Filter column M to see what was actually uploaded 
  • You'll also see the files in SuiteFiles 

With this test successfully completed you can be comfortable that the full migration will run. Delete the test content uploaded whenever you're ready.

Running the migration

Everything is now prepared to run the full migration. When doing this start out by:

  • Checking the properties of all files being uploaded and note the number of files, number of folders and volume of data 
  • Check out the number of items already in the SharePoint site by going to https://domain.sharepoint.com/_layouts/viewlsts.aspx and checking out the Shared Documents library. This is your starting point and once the folders are created you should have increased the item count by the number of folders detailed above (if the folder already exists in SuiteFiles it won't be created again so this can account for small differences in the numbers, also system folders are not created). 
  • Run the SuiteMoves.exe and drop in the new content for uploading 
  • The total number of files scanned should equal the number found in the file properties (above). The number of files selected for migration will be less than the total number of files scanned. This is due to system files that SharePoint won't accept. Click the SharePoint button 
  • The folders will be created. You can refresh the https://domain.sharepoint.com/_layouts/viewlsts.aspx page and see the item count increase as the folders are created. Check this count at the end of the process 
  • If concerned that the item count is incorrect refer to the latest finaloutput.csv, filter column C to TRUE to show all folders and then column J (IsInASystemFolder) also to TRUE. In this scenario 5 folders are listed in column X that start with ._ in the name that SharePoint will not accept. These are the 5 folders difference in the counts above. 
  • Run the SuiteMovesFinalUploadsTool.exe tool again using the new .migrationmodel file created until it says FINISHED. Click the Exporet Current CSV... button to generate the output report. 

Competing the migration

With this done the output.csv file needs examining. Open it, add the filters and inspect column M (Migration Status). You can see that some files have a status of FAIL:2 and Queued for upload..., this indicated that not all files were uploaded on the first run. The process at this point is to run the SuiteMovesFinalUploadsTool.exe tool again. The migration can become repetitive at this point if a large number of files still need uploading you may need to do this a number of times. Keep going until the only files left are those that have failed. You should be able to get all other files into SuiteFiles.

Some large files won't upload using the tool. Files in the 200-900MB range will need to be uploaded manually. This is done most efficiently by copying the value in column AA (UploadedLocation) and pasting the URL into the browser. This will open the SharePoint user interface for this location. Drag/drop the file into this location to upload it.

Things to know

You may find the following of use during a migration:

  • The SuiteMovesFinalUploadsTool.exe may stop (stall) during file upload process. If it does you can simply close the app and restart it. The tool will pick up where it left off.
  • SharePoint can throttle the upload process from time to time. The SuiteMovesFinalUploadsTool.exe tool will handle this and pause until the throttling is removed then start again.
  • If you have logged into the SuiteMovesFinalUploadsTool.exe but it stalls on a status of Getting Cookies this is an authentication issue, possibly you've entered the password incorrectly. Close the app and start again.
  • The SuiteMovesFinalUploadsTool.exe app can appear to not be doing a lot at the start, it may just say Uploading but the form doesn't refresh. Check the items counts in https://domain.sharepoint.com, if it's increasing the the app is just being slow to refresh.
  • The Jump To Active Item button takes you to the next item in the queue sequentially to upload.
  • The KICK ME button will take the item and push it to the top of the upload queue and force a result for it (upload or fail).

Resources

The following resources will be of use:

Still need help? Get in touch Get in touch