External Services Configuration

While this documentation focuses on self-managed, locally deployed services for maximum control and customization, MDSS supports any compatible external services. For cloud deployments, you can leverage managed services such as Amazon RDS, Amazon ElastiCache for Redis, Amazon MQ for RabbitMQ, Azure Database for PostgreSql, Azure Cache for Redis, Google Cloud Sql for PostgreSql, and Google Cloud Memorystore for Redis.

Note that some managed services may have compatibility limitations or require specific configuration adjustments compared to their open-source counterparts. The configuration principles outlined in this guide remain the same regardless of whether you choose self-managed or cloud-managed services - simply update the connection URLs and credentials in your customer.env file to point to your chosen service endpoints, ensuring the services support the required APIs and features that MDSS depends on.

Prerequisites

• Multiple VMs in the same network for MDSS deployment
• Separate VMs or managed services for external components
• Administrative access to all systems

Recommended specs:

• PostgreSQL: minimum 8 CPU and 16GB of RAM for the average workload
• RabbitMQ: RabbitMQ System Requirements
• Redis: Redis System Requirements
• MDSS VM: General System Requirements

1. External Components Setup

RabbitMQ Configuration

Installation: Follow the official RabbitMQ installation guide for your distribution.

Configure credentials for remote access:

sudo rabbitmqctl add_user <username> <password>

sudo rabbitmqctl set_permissions -p / <username> ".*" ".*" ".*"

sudo rabbitmqctl set_user_tags <username> administrator

Enable management interface for monitoring:

sudo rabbitmq-plugins enable rabbitmq_management

PostgreSQL Configuration

Installation: Follow the PostgreSQL installation guide.

Configurations:

1. Edit /etc/postgresql/*/main/postgresql.conf and set:

# Configure for remote access

listen_addresses = '*'

​

# PostgreSQL Performance Configuration

max_connections = 1000

​

# Memory settings for optimal database performance

shared_buffers = 16GB

work_mem = 256MB

temp_buffers = 64MB

maintenance_work_mem = 2GB

effective_cache_size = 24GB

​

# WAL (Write-Ahead Logging) configuration

wal_level = minimal

max_wal_senders = 0

max_replication_slots = 0

max_wal_size = 2GB

min_wal_size = 160MB

2. Increase file limits in sysctl.conf:

fs.file-max = 50000

3. Update systemd service file /usr/lib/systemd/system/postgresql.service to increase file limits for PostgreSQL user.
4. Restart PostgreSQL:

sudo systemctl daemon-reload && sudo systemctl restart postgresql.service

5. Set permissions for the public schema:

-- 1. Database connection

GRANT CONNECT ON DATABASE your_database TO <username>;

​

-- 2. Schema permissions

GRANT CREATE, USAGE ON SCHEMA public TO <username>;

​

-- 3. Full privileges on existing objects

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO <username>;

GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO <username>;

GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO <username>;

GRANT ALL PRIVILEGES ON ALL PROCEDURES IN SCHEMA public TO <username>;

​

-- 4. Default privileges for new objects

ALTER DEFAULT PRIVILEGES IN SCHEMA public 

GRANT ALL PRIVILEGES ON TABLES TO <username>;

​

ALTER DEFAULT PRIVILEGES IN SCHEMA public 

GRANT ALL PRIVILEGES ON SEQUENCES TO <username>;

​

ALTER DEFAULT PRIVILEGES IN SCHEMA public 

GRANT ALL PRIVILEGES ON FUNCTIONS TO <username>;

​

ALTER DEFAULT PRIVILEGES IN SCHEMA public 

GRANT ALL PRIVILEGES ON PROCEDURES TO <username>;

​

-- 5. For encryption extensions (if needed)

GRANT USAGE ON SCHEMA pg_catalog TO <username>;

Redis Configuration

Note: Use external Redis only when deploying MDSS services on multiple nodes. For single-node deployments, the local Redis instance provides better latency.

Installation: Follow the Redis installation guide.

Configure password and remote access in /etc/redis/redis.conf:

requirepass your_password_here

bind 0.0.0.0

2. MDSS Configuration

Configure External Services

Edit customer.env file with your external service configurations:

# Configure external RabbitMQ

RABBITMQ_URI=amqp://mdss:mdss@<rabbitmq-host>:5672

RABBITMQ_HOST=<rabbitmq-host>

RABBITMQ_PORT=5672

RABBITMQ_DEFAULT_PASS=mdss

RABBITMQ_DEFAULT_USER=mdss

​

# Optional: Configure external Redis instance 

CACHE_SERVICE_URI=<redis-host>:6379,user=default,password=mdss,syncTimeout=10000

CACHE_SERVICE_URL=<redis-host>

CACHE_SERVICE_PORT=6379

​

# Optional: Configure external MongoDB database (ony for data migration)

MONGO_URL=mongodb://<mongo-host>:27017/MDCS?maxPoolSize=50000

​

# Configure external PostgreSQL database

POSTGRESQL_URL=Host=<postgres-address>;Port=5432;Username=<username>;Password=<password>;Database=MDSS;Timeout=60;CommandTimeout=60

3. Performance Monitoring

Monitor RabbitMQ Queues

Access the RabbitMQ management interface at http://<rabbitmq-host>:15672 to monitor queue sizes.

Key queue to monitor:

• object_ready_for_scan_queue: Shows files waiting to be scanned

If this queue accumulates messages, consider increasing the number of scanning services.

Prefetch Count Optimization

When running multiple scanning services, decrease the scanning prefetch count to distribute load more evenly: RABBITMQ_SCANNING_PREFETCH_COUNT=5

Adjust other prefetch counts based on your queue sizes and performance requirements.

5. Best Practices

1. Resource Allocation: Ensure adequate CPU and memory for both MDSS and external service VMs
2. Network Latency: Keep all components in the same network to minimize latency
3. Monitoring: Regularly monitor RabbitMQ queues and system resources
4. Scaling: Start with fewer MDSS instances and scale up based on queue monitoring
5. Engine Selection: Use minimal engines required for your use case to improve performance

Troubleshooting

Common Issues:

• High queue buildup: Increase scanning service containers or optimize prefetch counts
• External service timeouts: Check network connectivity and service configuration
• Resource exhaustion: Monitor CPU/memory usage and scale resources accordingly

Log Locations:

• MDSS logs: /etc/mdss/services_logs/

This configuration enables horizontal scaling of MDSS deployments while maintaining shared state through external services, providing improved performance for high-volume scanning workloads.

Production Considerations

Security

Encrypted Connections: Always use secure connection protocols (TLS/SSL) for all communications between MDSS instances and external services. Configure RabbitMQ with AMQPS, and Redis with TLS encryption.

Network Security: Implement network-level security by restricting access to external services only from authorized MDSS hosts using firewalls, VPCs, or security groups. Deploy services in private subnets when possible.

Authentication & Authorization: Create dedicated service accounts with minimal required privileges for MDSS access.

High Availability

Configure external services for high availability using their native clustering or replication features:

• PostgreSQL: Use replication to a secondary database
• RabbitMQ: Configure clustering with HA policies
• Redis: Use Redis Sentinel or Redis Cluster

For managed cloud services, enable multi-AZ deployments and automatic failover capabilities provided by your cloud provider.

Monitoring & Alerting

Implement comprehensive monitoring for:

• Service availability and response times
• Resource utilization (CPU, memory, disk, network)
• Queue depths and processing rates
• Connection pool health
• Error rates and failed operations

Set up alerting for critical thresholds that could impact MDSS performance.

Backup & Disaster Recovery

Establish regular backup procedures and test recovery processes for all external services. For managed services, configure automated backups with appropriate retention policies.

Performance Optimization

• Configure connection pooling based on your concurrent load requirements
• Set appropriate resource limits and file descriptors
• Monitor and tune service-specific performance parameters
• Plan capacity based on expected MDSS throughput

Documentation References

For detailed configuration guidance, refer to the official production documentation:

• PostgreSQL Documentation
• RabbitMQ Production Checklist
• Redis Security Guidelines

For managed cloud services, consult your cloud provider's best practices documentation for the specific services you're using.