Deep CDR Performance (Throughput)

Performance Tuning Guide

Processing time and resources consumed by the engine vary relatively greatly depending on several aspects, e.g., nature and complexity of file structure, number of objects or components inside input files, etc. File size can be an aspect but is not necessary to contribute proportionally to processing time and resources consumption.

Depending on load and its complexity, users can tune some parameters to achieve a suitable balance on processing speed and throughput.

• Timeouts: a duration given to the engine for processing a file. Higher value gives the engine more chances to complete processing with complex files.

  • To set a value for this parameter, go to Workflow Management > Workflows > [Workflow name] > Deep CDR

• Parallel count: number of files being processed by the engine in parallel. Lower value puts low stress on the engine, giving it chances to complete processing with complex files.

  • Too large value can cause a domino effect that results in series of failures due to processing timed out.
  • Ones may keep it at 20 by default and, when necessary, should not increase it over 40.
  • To set a value for this parameter, see parallelcount_ds on MetaDefender Configuration.

• Number of Image Processing Threads: number of CPU cores being used for processing images. Higher value may drain CPU resources. Recommended value: <number of CPU cores>/8.

  • To set a value for this parameter, go to Inventory > Modules > Deep CDR.

Load with complex files

• Timeouts should be set to a high value, e.g., 30 minutes or 60 minutes.
• Parallel count should be set to a low value, e.g., 15 or 10.
• Number of Image Processing Threads should be set as recommended, <number of CPU cores>/8.

Load with less complex files

• Timeouts can be set to a low value, e.g., 5 minutes or 10 minutes.
• Parallel can be set to a high value, e.g., 20 but should not exceed 40.
• Number of Image Processing Threads should be set as recommended, <number of CPU cores>/8.

Throughput Test

Windows System Info

• RAM: 32.0 GB
• CPU: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
• OS: Window 10 x64
• Disk Drive: SSD 100.0 GB

Linux System Info

• RAM: 32 GB
• CPU: 16
• OS: CentOS Linux release 7.6.1810
• Disk Drive: 100 GB

Resources

• MetaDefender Core version

  • v5.x Windows: MetaDefender Core v5.0.0 with 8 engines
  • v5.x Linux: MetaDefender Core v5.0.0 with 10 engines

• Default Deep CDR configuration

  • Window version: 6.0.0.10522

Test Results