Active-Active configuration

This guide explains how to configure an active-active deployment of MDSS where multiple instances can use the same shared external services for improved performance and scalability.

Overview

In an active-active deployment, multiple MDSS instances work together using shared external components (RabbitMQ, PostgreSql, and Redis) to distribute workload, add redundancy and improve throughput. This configuration is particularly useful when handling large volumes of files.

External Services Deployment Options

While this documentation focuses on self-managed, locally deployed services for maximum control and customization, MDSS supports any compatible external services. For cloud deployments, you can leverage managed services such as Amazon RDS, Amazon ElastiCache for Redis, Amazon MQ for RabbitMQ, Azure Database for PostgreSql, Azure Cache for Redis, Google Cloud Sql for PostgreSql, and Google Cloud Memorystore for Redis.

Note that some managed services may have compatibility limitations or require specific configuration adjustments compared to their open-source counterparts. The configuration principles outlined in this guide remain the same regardless of whether you choose self-managed or cloud-managed services - simply update the connection URLs and credentials in your customer.env file to point to your chosen service endpoints, ensuring the services support the required APIs and features that MDSS depends on.

Prerequisites

• Multiple VMs in the same network for MDSS deployment
• Separate VMs or managed services for external components
• Administrative access to all systems

Recommended specs:

• MongoDB: MongoDB Hardware Requirements
• RabbitMQ: RabbitMQ System Requirements
• Redis: Redis System Requirements
• MDSS VM: General System Requirements

1. External Components Setup

RabbitMQ Configuration

Installation: Follow the official RabbitMQ installation guide for your distribution.

Configure credentials for remote access:

sudo rabbitmqctl add_user <username> <password>

sudo rabbitmqctl set_permissions -p / <username> ".*" ".*" ".*"

sudo rabbitmqctl set_user_tags <username> administrator

Enable management interface for monitoring:

sudo rabbitmq-plugins enable rabbitmq_management

MongoDB Configuration

Installation: Follow the MongoDB Community Edition installation guide.

Configure for remote access:

1. Edit /etc/mongod.conf and set:

net:

  bindIp: 0.0.0.0

2. Increase file limits in sysctl.conf:

fs.file-max = 50000

3. Update systemd service file /usr/lib/systemd/system/mongod.service to increase file limits for MongoDB user.
4. Restart MongoDB:

sudo systemctl daemon-reload && sudo systemctl restart mongod.service

Redis Configuration

Note: Use external Redis only when deploying MDSS services on multiple nodes. For single-node deployments, the local Redis instance provides better latency.

Installation: Follow the Redis installation guide.

Configure password and remote access in /etc/redis/redis.conf:

requirepass your_password_here

bind 0.0.0.0

2. MDSS Configuration

Configure External Services

Edit customer.env file with your external service configurations:

# Configure external RabbitMQ

RABBITMQ_URI=amqp://mdss:mdss@<rabbitmq-host>:5672

RABBITMQ_HOST=<rabbitmq-host>

RABBITMQ_PORT=5672

RABBITMQ_DEFAULT_PASS=mdss

RABBITMQ_DEFAULT_USER=mdss

​

# Optional: Configure external Redis instance 

CACHE_SERVICE_URI=<redis-host>:6379,user=default,password=mdss,syncTimeout=10000

CACHE_SERVICE_URL=<redis-host>

CACHE_SERVICE_PORT=6379

​

# Configure external database

MONGO_URL=mongodb://<mongo-host>:27017/MDCS?maxPoolSize=50000

​

# Example performance tuning parameters for higher workloads

RABBITMQ_SCANNING_PREFETCH_COUNT=5

RABBITMQ_LOAD_BALANCER_PREFETCH_COUNT=60

RABBITMQ_NFS_DISCOVERY_PREFETCH_COUNT=60

RABBITMQ_DISCOVERY_PREFETCH_COUNT=40

DISCOVERY_THROTTLER_MAX_PENDING_FILES=10000

3. Performance Monitoring

Monitor RabbitMQ Queues

Access the RabbitMQ management interface at http://<rabbitmq-host>:15672 to monitor queue sizes.

Key queue to monitor:

• object_ready_for_scan_queue: Shows files waiting to be scanned

If this queue accumulates messages, consider increasing the number of scanning services.

Prefetch Count Optimization

When running multiple scanning services, decrease the scanning prefetch count to distribute load more evenly: RABBITMQ_SCANNING_PREFETCH_COUNT=5

Adjust other prefetch counts based on your queue sizes and performance requirements.

5. Best Practices

1. Resource Allocation: Ensure adequate CPU and memory for both MDSS and external service VMs
2. Network Latency: Keep all components in the same network to minimize latency
3. Monitoring: Regularly monitor RabbitMQ queues and system resources
4. Scaling: Start with fewer MDSS instances and scale up based on queue monitoring
5. Engine Selection: Use minimal engines required for your use case to improve performance

Troubleshooting

Common Issues:

• High queue buildup: Increase scanning service containers or optimize prefetch counts
• External service timeouts: Check network connectivity and service configuration
• Resource exhaustion: Monitor CPU/memory usage and scale resources accordingly

Log Locations:

• MDSS logs: /etc/mdss/services_logs/

This configuration enables horizontal scaling of MDSS deployments while maintaining shared state through external services, providing improved performance for high-volume scanning workloads.

Production Considerations

Security

Encrypted Connections: Always use secure connection protocols (TLS/SSL) for all communications between MDSS instances and external services. Configure MongoDB with SSL, RabbitMQ with AMQPS, and Redis with TLS encryption.

Network Security: Implement network-level security by restricting access to external services only from authorized MDSS hosts using firewalls, VPCs, or security groups. Deploy services in private subnets when possible.

Authentication & Authorization: Create dedicated service accounts with minimal required privileges for MDSS access.

High Availability

Configure external services for high availability using their native clustering or replication features:

• MongoDB: Deploy as replica sets or sharded clusters
• RabbitMQ: Configure clustering with HA policies
• Redis: Use Redis Sentinel or Redis Cluster

For managed cloud services, enable multi-AZ deployments and automatic failover capabilities provided by your cloud provider.

Monitoring & Alerting

Implement comprehensive monitoring for:

• Service availability and response times
• Resource utilization (CPU, memory, disk, network)
• Queue depths and processing rates
• Connection pool health
• Error rates and failed operations

Set up alerting for critical thresholds that could impact MDSS performance.

Backup & Disaster Recovery

Establish regular backup procedures and test recovery processes for all external services. For managed services, configure automated backups with appropriate retention policies.

Performance Optimization

• Configure connection pooling based on your concurrent load requirements
• Set appropriate resource limits and file descriptors
• Monitor and tune service-specific performance parameters
• Plan capacity based on expected MDSS throughput

Documentation References

For detailed configuration guidance, refer to the official production documentation:

• MongoDB Production Notes
• RabbitMQ Production Checklist
• Redis Security Guidelines

For managed cloud services, consult your cloud provider's best practices documentation for the specific services you're using.