Overview

Parallel writes is a configurable feature that relates to the number of web service requests across an instance of DryvIQ on a given node, which will operate in parallel. However, it is essential to note that increasing the number of parallel writes does not always result in faster or better performance. There is a long list of concepts that have to be taken into account. Consult your Consultative Services representative or Customer Support for assistance understanding hw this setting can impact your jobs and configuration. Also, if you are running DryvIQ in an auto-scale environment, this does not apply as scaling will occur as configured.

Parallel Writes and Memory Usage

A job in DryvIQ does not use a fixed amount of memory. Memory usage for individual jobs will vary based on several factors, the most significant being the number of files and how they are distributed (all in one folder, spread across subfolders, etc.). To avoid excessive memory usage related to how content is distributed, DryvIQ recommends preserving the system default for Directory Item Limits. The main factors affecting memory usage for a DryvIQ node are the number of concurrent jobs, the Parallel Writes Per Job for each job, and the memory impact of the specific jobs.

Addressing Memory Issues

If memory issues occur due to increasing the Directory Item Limit or Parallel Writes Per Job, there is no other mitigation other than reducing the number of current jobs or breaking up the source content into multiple jobs. DryvIQ will continue using memory until it runs out (it will not limit itself), and it will eventually reach the environment's maximum capacity. Reaching the environment maximum may result in a non-graceful termination of DryvIQ, which could cause jobs to retransfer files, permissions, or metadata. In the case of larger jobs being stopped in this manner, they will enter recovery mode, continue to use all available memory, stop again, and repeat the process in a loop, resulting in a loss of throughput.

Default Parallel Write Settings

The default parallel write value is 4, 8, or 12, depending on the number of the CPU logical processors on the machine running the DryvIQ service.

• If the CPU Logical Processors is 2, the default parallel writes value is 4.
• If the CPU Logical Processors is 8, the default parallel writes value is 8.
• If the CPU Logical Processors is 32, the default parallel writes value is 12.

Set Parallel Writes for a Job

Add the following in the transfer block of your job.

{
 "performance": {
       "parallel_writes": {"requested": 4}
    }
}

Example

{
    "name": "Test Parallel Writes",
        "kind": "transfer",
        "transfer": {
            "audit_level": "trace",
            "transfer_type": "copy",
            "performance": {
                "parallel_writes": {"requested": 4}
},
    "source": {
        "connection": {
            "id": "{{cloud_connection}}"
},
    "target": {
        "path": "/MASTER_TESTS/BASIC TRANSFER TESTS"
    }
},
    "destination": {
        "connection": {
            "id": "{{cloud_connection}}"
},
    "target": {
        "path": "/SAP/LB/Test_ParallelWrites"
        }
    }
},
    "schedule": {
        "mode": "manual"
    }
}

To review your job to confirm your request for parallel writes, use the following call.

GET {{url}}v1/jobs?include=all

Update Parallel Writes on an Existing Job

The following body in a PATCH request to {{url}}v1/jobs/{{job}} will update the parallel_writes value to 8.

{
   "kind": "transfer",
    "transfer": {
     "performance": {
           "parallel_writes": {"requested": 8}
     }
   }
}