We’re excited to introduce MetaDefender Distributed Cluster - a high-performance deployment architecture for MetaDefender Core, designed to deliver scalable throughput, fault tolerance, and centralized control in high-volume file security environments.
Available in MetaDefender Core version 5.14.2 and above, this deployment model enables organizations to optimize how they handle large-scale file uploads, distributed processing, and multi-instance scanning, while maintaining full operational visibility across all deployed components.
Designed to meet the demands of enterprise and mission-critical use cases, MetaDefender Distributed Cluster helps security teams streamline operations without sacrificing performance, availability, or control.
Built for Performance at Scale
As file volumes grow and operational complexity increases, organizations need more than just powerful scanning engines; they need a deployment architecture that scales seamlessly, balances workloads intelligently, and delivers consistent performance under pressure.
MetaDefender Distributed Cluster addresses these challenges with a modular, distributed design that supports:
- Horizontal scaling across instances and regions
- Parallelized processing of complex file types and nested archives
- High availability and fault tolerance for mission-critical environments
- Centralized orchestration and monitoring for operational control
This deployment option is ideal for customers operating in high-throughput, multi-tenant, or geographically distributed infrastructures.
System Architecture Overview
MetaDefender Distributed Cluster separates responsibilities across specialized components to enable scalable routing, storage, scanning, and management:
- API Gateway: Accepts and routes file scan requests from clients
- File Storage: Securely stores submitted and processed files
- Worker: Manages the health and orchestration of deployed components
- Control Center: Web UI for deployment, upgrades, configuration, and monitoring
- Identity Service: User authentication and authorization
- MetaDefender Core: Executes file sanitization, vulnerability assessment, and multi-engine scanning
- RabbitMQ: Queues tasks and distributes them efficiently to MetaDefender Core instances
- Redis: Provides in-memory storage for fast access to intermediate scan results
Key Capabilities
The MetaDefender Distributed Cluster deployment includes the full capabilities of MetaDefender Core, enhanced with distributed orchestration and control.
Centralized Orchestration and Management
Manage all MetaDefender Core instances and API Gateways from a single Control Center.
- Deploy and upgrade MetaDefender Core and API Gateway instances
- Manage licenses, engine installers, certificates, and workflows
- Monitor system health, performance metrics, scan history, audit logs, and executive reports
- Enable secure communication via HTTPS for the Control Center and API Gateway, with all other components communicating securely by default using built-in SSL certificates
Advanced API Capabilities
Automate and integrate file scanning across your infrastructure:
- Submit files or batches for scanning
- Retrieve scan results by data_id or file hash
- Download sanitized or original files
- Cancel in-progress scans
- Retrieve engine versions and scanning rules
High-Volume File Processing
Built for throughput and efficiency:
- Distribute scanning tasks across MetaDefender Core instances
- Parallelize archive extraction and scanning to reduce processing time
- Use RabbitMQ to balance workload dynamically
- Handle nested or complex archives with faster turnaround
Key Benefits
MetaDefender Distributed Cluster is optimized for organizations operating at scale. It delivers:
High Availability
Distributed architecture ensures continuous inspection, even during instance-level outages or maintenance.
Fault Tolerance
Designed to isolate failures and support automatic recovery, preserving uptime and service continuity.
Load Distribution across MetaDefender Core Instances
Share file scanning tasks across multiple instances, eliminating performance bottlenecks.
Parallel Archive File Handling
Extract and distribute archive contents across MetaDefender Core instances, significantly reducing scan duration for large files.
Elastic Scalability
Add more instances as file volumes grow, scaling capacity seamlessly to meet organizational demand.
Centralized Orchestration
Manage all MetaDefender Core instances and API Gateways from a single Control Center web console. Provide a zero-downtime upgrade mechanism with just a few simple operations.
Performance Benchmark Report
A performance evaluation was conducted to analyze how MetaDefender Distributed Cluster handles varying file sizes and file types. The benchmark demonstrates how performance scales with workload complexity, particularly when processing large files, nested archives, or formats requiring intensive inspection.
Data Set
| File Security | File Type | Number of Files | Total Size | Average File Size |
|---|---|---|---|---|
| Medium archive files | RPM CAB EXE | 50 | Compressed size: 2.8 GB Extracted size: 12.09 GB | Compressed size: 56.02 MB Extracted size: 0.036 MB |
| Big archive files | CAB | 4 | Compressed size: 2.9 GB Extracted size: 124 GB | Compressed size: 715 MB |
Performance Test Result
Scenario 1
Submit 400 medium archive files to MetaDefender Distributed Cluster at a rate of 1 file per second.
Scenario 2
Submit 4 large CAB files to MetaDefender Distributed Cluster.
The scenarios replicate 02 different routing cases of a common Load Balancer (LB).
- LB OneToOne: An ideal routing ensures that one CAB file is routed to a single MD Core.
- LB FourToOne: The worst routing that delivered four CAB files to a single MD Core.
See more performance results
Choosing the Right Deployment Option
MetaDefender Core is designed to offer flexibility in how customers deploy it based on their infrastructure, operational needs, and scalability requirements. The following overview highlights available deployment models:
| Deployment Option | Best for |
|---|---|
| API-based | Environments with predictable file volumes and centralized operations |
| Cloud | Teams looking for managed infrastructure and easy integration in the cloud |
| Cloud image | Pre-configured templates to launch MetaDefender Core instance in the cloud including AWS, Azure, and GCP for quick setup and scanning |
| Containerized | Kubernetes-native or containerized workflows with dynamic scaling needs |
| Distributed Cluster | Large-scale, high-availability environments requiring parallel processing and centralized control |
Each option is designed to deliver the same industry-leading file security capabilities, with different operational advantages based on your architecture.
Next Steps
MetaDefender Distributed Cluster offers the flexibility and scalability that modern security infrastructures demand, whether you're scanning millions of files daily, operating across multiple data centers, or building a resilient security architecture for mission-critical environments.
MetaDefender Distributed Cluster is now available for MetaDefender Core version 5.14.2 and above. To get started:
- Visit: opswat.com/products/metadefender/core
- View the https://www.opswat.com/docs/mdcore/distributed-cluster-deployment in Docs
- Need help? Contact: support@opswat.com
