Data Migration - Loading Documents

Overview

The loading of documents consists primarily of two activities: loading the documents as files into the destination DMS and providing metadata to ShareDo for task creation and chronology links.

The documents can be uploaded via an upload tool that will orchestrate the import from source to target.

ShareDo (meta)data is optional and enables links from ‘Prepare Document Tasks’ and the case's Chronology to the specific file within the DMS. The link at the case level to the DMS is established as part of case data ETL (and is a manual task depending on the DMS used).

Document Content

The files can be organised in a variety of ways, though the most common is a folder structure of one folder per case and sub-folders to match the DMS. An upload tool will take the files from the staging area and copy them to the DMS. For example, ShareGate Desktop is an excellent tool for uploading files from a filesystem into SharePoint.

While the method of control of the upload tool will vary, using ShareGate Desktop as an example:

Sites are created in SharePoint, one per case
Files are staged in a filesystem
File data is provided detailing file paths, owner, destination folder, creation dates and document title
CSV files are generated, one per case, with the attributes of each file, the target SharePoint site & folder
ShareGate Desktop is used to upload the files into SharePoint, processing one CSV for one case’s documents
Repeat step 5 for all cases

Document loading is independent of the ETL data load. Depending on the volume of documents, the runtime may take days (or even weeks). Typically, files don’t change once loaded, so it is possible to start the document load prior to the main 'go-live' event.

ShareDo can assist with the document loading, however, it is typically a client-led activity.

Document Metadata

Loading document metadata is a part of the normal dataload process and not dependent on the files being available. The metadata is simply the data held within ShareDo’s database, which provides links into the DMS at a file level. Files can exist within the DMS without explicit links from ShareDo - they would be visible from the DMS view that links at the case level.

When specifying documents in the loader, there are two variations:

Whether the document meta should be simply recorded against the Case, in which case a chronology event will be raised.
Whether the document should be loaded in the context of a new Task or Prepare Document style activity. Optionally this can further include further attributes such as postage, recorded delivery reference, etc, that are recorded in an associated child task (a complex scenario).

The simplest and most efficient method is to load the document references as Chronology events, when using tasks as the context load rates are slower.

Name	Type	Mandatory	Notes
dataLoadId	Uniqueidentifier	Yes	The load this record is associated with
documentLoadType	Nvarchar	Yes	Specifies the type of load that should be performed. Valid values are TaskRef – document is to be loaded against an existing task TaskStandalone – document is to be loaded and a task created as a single action Chronology – document is uploaded as a chronology record
documentReference	Nvarchar	Yes	Unique reference for this document in the context of ETL
ShareDoReference	Nvarchar	Yes	The primary case that this document refers to. This is typically a Case.
title	Nvarchar	Yes	Title
documentId	Nvarchar	Yes	Link into the DMS (see notes below)
createdDate	Datetime	Yes	Date the document was created
ownerUserReference	Nvarchar	No	Link to the user in import.users to record as the owner (chronology and for tasks)
primaryShareDoTaskRef	Nvarchar	No	If provided, will create a link from the existing task in ShareDo to the document
Fields required when creating a chronology record as part of the document upload
poiSystemName	Nvarchar	Yes	System name of the Chronology POI type. We currently support the following POI Handlers document-filing document-upload docgen (planned change) NOTE: We could potentially support an email-upload fairly easily.
Fields required when creating a task as part of the document upload
ShareDoTypeSystemName	Nvarchar	Yes	System name of the task type – this should typically be a prepare document
phaseSystemName	Nvarchar	Yes	Current phase of the task created
category	Nvarchar		ShareDo allows you to optionally specify the sub categorisation of a ShareDo. Since not all ShareDo types require a category you will need to determine if the ShareDo type requires a category and if so what values it contains. You can see if the ShareDo type is configured for a category by inspecting the column `categoryOptionSetSystemName` returned in the query above. The valid optionset values are then contained within the table ods.optionSetValues or can be viewed within the application.The `categoryId` column is data loaded and requires to be set with the `optionId` of the desired category. `optionID` is obtianed from `[sharedo_import].[importConfig].[sharedoTypesCategories].[optionId]` The `category` column isn't data loaded and is provided as a location to store a human readable description of the category set in the `categoryId` column. This is to enable better readability of the data in the `[sharedo_import].[source].[sharedo]` table.
Title			The title of the ShareDo that is shown to the user in lists and forms
Description			The description or details of the ShareDo that is shown to the user in the various details
Additional fields required when creating a delivery method associated with the PrepDoc task
Owner			Owner of the task
RecordDeliveryFlag	Boolean		If this is set then a delivery method should be recorded
DeliveryMethodSystemName	Nvarchar	Yes – if RecordDelivery	Valid delivery method for the task. Valid delivery methods are such as: - task-activity-print-post - task-activity-send-email Delivery methods vary per implementation
Postage Class		Yes – if RecordDelivery	Valid postal methods are specified in the optionset - print-room-postage-method - Please use meaning code as the value
Tracing Reference	NVarchar		The postage tracking reference

Example Execution Plan Using SharePoint

A typical split of tasks and ownership would be:

Client will:

Provision a Windows server with sufficient disk space to hold all documents
Stage the files into the holding area
Provide site details (via database table) - site unique ID, site title
Provide document details (via database table) - filename, path in staging area, author, title, create date

ShareDo will:

Create sites, one per case (if on-premises SharePoint, then also create site collections according to volumes)
Prepare CSV files using provided data for ShareGate Desktop (client license)
Execute document load using ShareGate Desktop
Collate upload results (via database table) and report back to the client

Overview

Document Content

Document Metadata

Example Execution Plan Using SharePoint

Related Articles