Overview

While all platforms do not impose limits on the number of items that can be stored in a single directory, their APIs may limit the number of items that can be downloaded or uploaded. This means that when DryvIQ attempts to download content from the source that exceeds the platform API limit, errors may start to occur. The DryvIQ Platform engine also has a limit on the number of items it can crawl and download from a single directory, based on the environment setup, such as memory allocation.

A job in DryvIQ does not use a fixed amount of memory. Memory usage for individual jobs will vary based on several factors, the most significant of which is the number of files and how they are organized (all in one folder, spread across subfolders, etc.). The main factors affecting memory usage for a DryvIQ node will be the number of concurrent jobs, the Parallel Writes Per Job for each job, and the memory impact of the specific jobs.

Based on source and destination platform limits and potential excessive memory usage, the DryvIQ system configuration defaults to a maximum of 10,000 items per container to ensure a successful transfer.

Recommendations

To avoid these errors, DryvIQ recommends configuring the directory item limit policy for your job. This option is configurable, with a default system maximum of 10,000 for a single directory, which corresponds to most platforms' recommendations. While some platforms, such as Box, allow up to 15,000 items in a single directory, changing this configuration limit may result in reduced performance and potentially unexpected errors. To improve the performance of your transfer, DryvIQ recommends using the DryvIQ system configuration of a maximum of 10,000 items per container. DryvIQ will identify directories that exceed this limit and flag them.

When viewing the job's Log page in the user interface, these items will be flagged as “Maximum number of items exceeded.” 

In the log files, these items will be flagged with

• "Level": "Error",
• "Name": "TransferExecutor",
• "Message": “The path exceeds the maximum number of 10,000 children: /path.” (Note that the number of children will reflect the maximum number set if you edited the default limit.)

During remediation, evaluate the source content and move items into sub-folders until each directory contains 10,000 items or fewer.

What can happen when increasing the Directory Limit?

DryvIQ strongly recommends using the default directory limit of 10,000. Increasing this value can have the following impacts:

• Performance issues with excessive memory usage
• Limit errors from the source and destination platforms that cause failures in your DryvIQ job. (See Platform Limit documentation references below.)

Addressing Memory Issues

If memory issues occur due to increasing the Directory Item Limit or Parallel Writes Per Job, there is no other mitigation other than reducing the number of current jobs or breaking up the source content into multiple jobs. DryvIQ will continue using memory until it runs out (it does not self-limit), and it will eventually reach the environment's maximum. Reaching the environment maximum may result in a non-graceful termination of DryvIQ, which could cause jobs to retransfer files, permissions, or metadata. In the case of larger jobs being stopped in this manner, they will enter recovery mode, continue to use all available memory, stop again, and repeat this loop, causing a loss of throughput.

Expected Behavior

The parent folder will transfer to the destination. Permissions will be applied, and the job will have failures with the following log:

"Other activity | Error activity

The path exceeds the maximum number of 10,000 children: /path"

Default Behavior

If "max_items_per_container" is not configured in the job JSON, the default limit of 10,000 will still apply.

Change Limit When Creating a Job

See “Additional Configuration Options” for Scripting.

Configuration Options in the appSettings.json

See Configuration Options. The default value can also be changed using "transfers:max_items_per_container" config key.

Reporting in The User Interface

The “Limit Transfer” error will be displayed on the overview.

Platform Limitation

If you configure DryvIQ above 10,000, please refer to the documentation for your source and destination platforms for potential restrictions on those platforms.

Job Configuration Through Rest API

{
    "name":"Example Job Configuration - Max Items Per Container",
    "kind": "transfer",
    "transfer": {
      "transfer_type": "copy",
      "audit_level": "trace",
      "batch_mode": "always",
      "conflict_resolution": "latest",
      "delete_propagation": "ignore_both",
      "failure_policy": "continue",
      "large_item": "skip",
      "lock_propagation": "ignore",
      "max_items_per_container": 10000,
      "performance": {
                "parallel_writes": {
                    "requested": 4
                }
      },          
      "permissions": {
    		"policy": "add",
    		"links": true,
    		"failures": "exceptions"
      },
      "preserve_owners": true,
      "timestamps": true,
      "empty_containers": "create",
      "duplicate_names": "rename",
      "item_overwrite": "overwrite",
      "restricted_content": "convert",
      "segment_transform": true,
      "versioning": {
    		"preserve": "native",
    		"select": "all"
      },
         "group_map": {
         "id": "{{group_map_id}}",
         "type": "group_map"
      },
      "account_map": {
         "id": "{{account_map_id}}",
         "type": "account_map"
    },
        "filter":{
         "source":[
            {
               "action":"exclude",
               "rules":[
                  {
                     "type":"filter_shared"
                  }
               ],
               "type":"filter_rule"
            }
         ],
         "destination":[
            {
               "action":"exclude",
               "rules":[
                  {
                     "type":"filter_shared"
                  }
               ],
               "type":"filter_rule"
            }
         ]
        },        
        "source": {
            "connection": { "id": "{{cloud_connection_source}}" },
            "impersonate_as": { "email": "jsmith@yourdomain.com" },
            "target": {
                "path": "/SourcePath"
            }
        },
        "destination": {
            "connection": { "id": "{{cloud_connection_destination}}" },
            "impersonate_as": { "email": "jsmith@mydomain.onmicrosoft.com" },
            "target": {
                "path": "/DestinationPath"
            }
        },
        "simulation_mode": false
    },
    "schedule": {
        "mode": "manual"
    },
    "stop_policy": {
        "on_success": 5,
        "on_failure": 5,
        "on_execute": 25
    },
    "category": {
      "name": "Category 1"
    }
}