Bulk Migration (Batch Mode) Using REST API
Learn how to use batch mode for migration jobs using the DryvIQ REST API.
Table of Contents
Overview
Batch mode is an optimized way to upload a high volume of data using a special API provided by some platforms. Currently, Dropbox, Dropbox for Business, Microsoft Office 365, and Microsoft OneDrive for Business have support for this feature. When running in batch mode, instead of committing one file at a time, files are committed in batches optimized for each platform.
To leverage the Bulk (Batch) Migration API, transfer accounts must be an Site Collection Administrator. Non-Administrator transfer accounts will default to use CSOM, even when batch mode is requested with the job.
Configure Your Job
Setting transfer.batch_mode to one of the following options can change or disable the batch mode policy.
Options | Behavior |
---|---|
none | Disable the batch mode for all job's executions |
initial | Use batch mode for the first job execution only |
always | Use batch mode for all job executions (default behavior) |
{
"name":"Simple Transfer Job with no batch mode",
"kind": "transfer",
"transfer": {
"batch_mode": "none",
"transfer_type": "copy",
"source": {
"connection": { "id": "{{nfs_connection}}" }
},
"destination": {
"connection": { "id": "{{cloud_connection}}" },
"target": {
"path": "/MyDestination1"
}
}
},
"schedule": {
"mode": "manual"
}
}
Enable Batch Manifests
Batch Manifests are configured in your appSettings.json file.
- You need the platform ID. See Connection Management for the call to return the platform ID.
- You must restart your DryvIQ Service to enable the batch manifest feature.
- Batch manifests are located in your log directory.
This element must be included to enable batch manifests. Replace "platform_id" with the corresponding platform ID.
}
"platform_id": {
"batch_save_manifests": true
}
The example code below shows enabling batch manifest for Office 365 (platform_id = office365-graph) and OneDrive for Business (platform ID = onedrive-business-graph).
{
"data": {
"provider": "sqlserver",
"connection": "SERVER=localhost;DATABASE=SkySyncV4;Integrated Security=SSPI;",
"directory": "C:\\ProgramData\\SkySync\\v4"
},
"server": {
"proxy_url": "http://192.168.1.76:9090/"
},
"deployment": {
"packageDirectory": "C:\\ProgramData"
},
"office365-graph": {
"batch_save_manifests": true
},
"onedrive-business-graph": {
"batch_save_manifests": true
},
"license": {
"activation_key": "ab4e994a-342f-0000-0000-b5da2577579b"
}
}
}
Configure Batch Size
The Batch Size is configured in your appSettings.json file.
- You need the platform ID. See Connection Management for the call to return the platform ID.
- You must restart your DryvIQ Service for the batch size to be applied.
- The Default Batch Size is 100.
{
"data": {
"provider": "sqlserver",
"connection": "SERVER=localhost;DATABASE=SkySyncV4;Integrated Security=SSPI;",
"directory": "C:\\ProgramData\\SkySync\\v4"
},
"server": {
"proxy_url": "http://192.168.1.76:9090/"
},
"deployment": {
"packageDirectory": "C:\\ProgramData"
},
"office365": {
"batch_save_manifests": true,
"batch_item_limit": "5"
},
"onedrive-business": {
"batch_save_manifests": true,
"batch_item_limit": "5"
},
"license": {
"activation_key": "ab4e994a-342f-0000-0000-b5da2577579b"
}
}
}
Setting Through the Scripting Page in the UI
Add the following to the JSON block on the Scripting page in the Job Advanced Features when creating the job.
{
"transfer": {
"batch_mode": "none"
}
}