External Services Configuration

While this documentation focuses on self-managed, locally deployed services for maximum control and customization, MDSS supports any compatible external services — whether self-managed on your own VMs or consumed as managed services from a Cloud Service Provider (CSP).

Protocol Compatibility

MDSS communicates with its external stateful dependencies using standard open-source protocols only:

• PostgreSQL: standard PostgreSQL wire protocol

• RabbitMQ: AMQP 0-9-1 (the standard RabbitMQ protocol)

• Redis: standard Redis protocol (RESP)

Any service — managed or self-deployed — that exposes these standard protocols is compatible with MDSS. This is why managed services not explicitly listed in this guide (for example, managed PostgreSQL on Oracle Cloud) also work without compatibility issues. Conversely, managed services that only expose proprietary or different protocols (for example, Azure Service Bus, which does not implement AMQP 0-9-1 semantics compatible with RabbitMQ clients) are not compatible.

If your cloud provider does not offer a protocol-compatible managed service for a given component, either:

• Self-deploy the service on a Linux VM in your cloud environment, following the External Components Setup instructions below, or

• Use a CSP marketplace alternative — for example, CloudAMQP provides managed RabbitMQ available through the Azure Marketplace and Google Cloud Marketplace, and can be recommended across any CSP.

The configuration principles outlined in this guide remain the same regardless of whether you choose self-managed or cloud-managed services — simply update the connection URLs and credentials in your customer.env file to point to your chosen service endpoints.

Supported Managed Services by Cloud Provider

PostgreSQL

Amazon RDS for PostgreSQL

1. In the AWS Console, create an RDS for PostgreSQL instance, choosing an instance class that meets the recommended specs.

2. Deploy it in the same VPC as your MDSS hosts and configure the security group to allow inbound traffic on port 5432 from the MDSS hosts only.

3. Connect with the master user and create the mdss user, the MDSS database, and grants as described in PostgreSQL Configuration (steps 1–2).

4. Copy the instance endpoint into POSTGRESQL_URL in customer.env on the MDSS host and disable the embedded PostgreSQL container (DISABLED_SERVICES=postgres).

Reference: Creating and connecting to a PostgreSQL DB instance (AWS)

Azure Database for PostgreSQL

1. In the Azure Portal, create an Azure Database for PostgreSQL – Flexible Server instance sized per the recommended specs.

2. Use private access (VNet integration) or configure firewall rules so only MDSS hosts can reach port 5432.

3. Connect with the admin user and create the mdss user, the MDSS database, and grants as described in PostgreSQL Configuration (steps 1–2).

4. Copy the server FQDN into POSTGRESQL_URL in customer.env and disable the embedded PostgreSQL container (DISABLED_SERVICES=postgres).

Reference: Create an Azure Database for PostgreSQL flexible server (Microsoft)

Google Cloud SQL for PostgreSQL

1. In the Google Cloud Console, create a Cloud SQL for PostgreSQL instance sized per the recommended specs.

2. Enable a private IP on the VPC shared with your MDSS hosts (or use authorized networks) so only MDSS hosts can reach port 5432.

3. Connect with the default postgres user and create the mdss user, the MDSS database, and grants as described in PostgreSQL Configuration (steps 1–2).

4. Copy the instance IP into POSTGRESQL_URL in customer.env and disable the embedded PostgreSQL container (DISABLED_SERVICES=postgres).

Reference: Create a Cloud SQL for PostgreSQL instance (Google)

RabbitMQ

Amazon MQ for RabbitMQ (AWS)

1. In the AWS Console, create an Amazon MQ broker with the RabbitMQ engine type, deployed in the same VPC as your MDSS hosts.

2. Configure the broker's security group to allow inbound traffic on port 5671 (AMQPS) from the MDSS hosts.

3. Create the broker admin credentials and note the AMQPS endpoint.

4. Set RABBITMQ_URI (using the amqps:// scheme), RABBITMQ_HOST, RABBITMQ_PORT, RABBITMQ_DEFAULT_USER, and RABBITMQ_DEFAULT_PASS in customer.env, and disable the embedded RabbitMQ container (DISABLED_SERVICES=rabbitmq).

5. Review the broker's consumer timeout configuration — MDSS long-running scan consumers require the consumer timeout to be disabled or increased, as described in RabbitMQ Configuration.

Reference: Getting started with Amazon MQ for RabbitMQ (AWS)

CloudAMQP (Azure and Google Cloud)

Azure and Google Cloud do not offer a native managed service compatible with the standard RabbitMQ (AMQP 0-9-1) protocol — Azure Service Bus is not RabbitMQ-compatible. For these providers, use CloudAMQP (managed RabbitMQ available via the CSP marketplaces) or self-deploy RabbitMQ on a Linux VM as described in RabbitMQ Configuration.

1. Subscribe to CloudAMQP through the Azure Marketplace or Google Cloud Marketplace, or sign up directly at cloudamqp.com.

2. Create an instance in the same cloud region as your MDSS deployment; for production, choose a dedicated plan and enable VPC peering to keep traffic private.

3. From the instance details page, copy the AMQPS URL and credentials.

4. Set RABBITMQ_URI (using the amqps:// scheme, including the vhost from the CloudAMQP URL), RABBITMQ_HOST, RABBITMQ_PORT, RABBITMQ_DEFAULT_USER, and RABBITMQ_DEFAULT_PASS in customer.env, and disable the embedded RabbitMQ container (DISABLED_SERVICES=rabbitmq).

5. Disable the consumer timeout for the instance (configurable via the CloudAMQP console) as described in RabbitMQ Configuration.

Reference: CloudAMQP documentation

Self-deployed RabbitMQ on a CSP Linux VM (any provider)

If a marketplace service is not an option, provision a Linux VM (e.g., Azure Virtual Machines, Google Compute Engine, Amazon EC2) in the same network as your MDSS hosts and follow the RabbitMQ Configuration instructions below.

Redis

Amazon ElastiCache for Redis (AWS)

1. In the AWS Console, create an ElastiCache for Redis cluster in the same VPC as your MDSS hosts.

2. Configure the security group to allow inbound traffic on port 6379 from the MDSS hosts.

3. Enable authentication (Redis AUTH) and, for production, in-transit encryption (TLS).

4. Copy the primary endpoint into CACHE_SERVICE_URI, CACHE_SERVICE_URL, and CACHE_SERVICE_PORT in customer.env, and disable the embedded Redis container (DISABLED_SERVICES=redis).

Reference: Getting started with Amazon ElastiCache (AWS)

Azure Cache for Redis

1. In the Azure Portal, create an Azure Cache for Redis instance in the same region as your MDSS deployment.

2. Use private endpoint / VNet integration or firewall rules so only MDSS hosts can reach the cache.

3. From Access keys, copy the host name, port, and access key.

4. Set CACHE_SERVICE_URI (host, port, and the access key as the password), CACHE_SERVICE_URL, and CACHE_SERVICE_PORT in customer.env, and disable the embedded Redis container (DISABLED_SERVICES=redis).

Reference: Create an Azure Cache for Redis instance (Microsoft)

Google Cloud Memorystore for Redis

1. In the Google Cloud Console, create a Memorystore for Redis instance on the VPC shared with your MDSS hosts.

2. Enable AUTH and, for production, in-transit encryption (TLS).

3. Note the instance IP address, port, and AUTH string.

4. Set CACHE_SERVICE_URI, CACHE_SERVICE_URL, and CACHE_SERVICE_PORT in customer.env accordingly, and disable the embedded Redis container (DISABLED_SERVICES=redis).

Reference: Create a Memorystore for Redis instance (Google)

Prerequisites

• Multiple VMs in the same network for MDSS deployment

• Separate VMs or managed services for external components

• Administrative access to all systems

Recommended specs:

• PostgreSQL: minimum 8 CPU and 16GB of RAM for the average workload

• RabbitMQ: RabbitMQ System Requirements

• Redis: Redis System Requirements

• MDSS VM: General System Requirements

1. External Components Setup

RabbitMQ Configuration

Installation: Follow the official RabbitMQ installation guide for your distribution.

Configure credentials for remote access:

Enable management interface for monitoring:

Disable consumer timeout (important for long processing consumers)

Edit RabbitMQ config:

Add:

Restart RabbitMQ

PostgreSQL Configuration

Installation: Follow the PostgreSQL installation guide.

Configure for remote access:

1. Create the Database User and Database

Connect to PostgreSQL as a superuser and run:

2. ** Grant Required Permissions

**

Still connected as superuser, switch to the MDSS database and run:

3. Configure postgresql.conf:

4. Increase file limits in sysctl.conf:

5. Update systemd service file /usr/lib/systemd/system/postgresql.service to increase file limits for PostgreSQL user.

6. Restart PostgreSQL:

7. Configure customer.env on the MDSS Host

On the MDSS host, edit /etc/mdss/customer.env (in your MDSS installation directory) and add:

Replace <pg-host-ip> with the IP or hostname of your PostgreSQL server and <your_password> with the password you set in Step 1.

8. Disable the Embedded PostgreSQL Container

Add the following to customer.env:

Redis Configuration

Note: Use external Redis only when deploying MDSS services on multiple nodes. For single-node deployments, the local Redis instance provides better latency.

Installation: Follow the Redis installation guide.

Configure password and remote access in /etc/redis/redis.conf:

2. MDSS Configuration

Configure External Services

Edit customer.env file with your external service configurations:

3. Performance Monitoring

Monitor RabbitMQ Queues

Access the RabbitMQ management interface at http://<rabbitmq-host>:15672 to monitor queue sizes.

Key queue to monitor:

• object_ready_for_scan_queue: Shows files waiting to be scanned

If this queue accumulates messages, consider increasing the number of scanning services.

Prefetch Count Optimization

When running multiple scanning services, decrease the scanning prefetch count to distribute load more evenly: RABBITMQ_SCANNING_PREFETCH_COUNT=5

Adjust other prefetch counts based on your queue sizes and performance requirements.

5. Best Practices

1. Resource Allocation: Ensure adequate CPU and memory for both MDSS and external service VMs

2. Network Latency: Keep all components in the same network to minimize latency

3. Monitoring: Regularly monitor RabbitMQ queues and system resources

4. Scaling: Start with fewer MDSS instances and scale up based on queue monitoring

5. Engine Selection: Use minimal engines required for your use case to improve performance

Troubleshooting

Common Issues:

• High queue buildup: Increase scanning service containers or optimize prefetch counts

• External service timeouts: Check network connectivity and service configuration

• Resource exhaustion: Monitor CPU/memory usage and scale resources accordingly

Log Locations:

• MDSS logs: /etc/mdss/services_logs/

This configuration enables horizontal scaling of MDSS deployments while maintaining shared state through external services, providing improved performance for high-volume scanning workloads.

Production Considerations

Security

Encrypted Connections: Always use secure connection protocols (TLS/SSL) for all communications between MDSS instances and external services. Configure RabbitMQ with AMQPS, and Redis with TLS encryption.

Network Security: Implement network-level security by restricting access to external services only from authorized MDSS hosts using firewalls, VPCs, or security groups. Deploy services in private subnets when possible.

Authentication & Authorization: Create dedicated service accounts with minimal required privileges for MDSS access.

High Availability

Configure external services for high availability using their native clustering or replication features:

• PostgreSQL: Use replication to a secondary database

• RabbitMQ: Configure clustering with HA policies

• Redis: Use Redis Sentinel or Redis Cluster

For managed cloud services, enable multi-AZ deployments and automatic failover capabilities provided by your cloud provider.

Monitoring & Alerting

Implement comprehensive monitoring for:

• Service availability and response times

• Resource utilization (CPU, memory, disk, network)

• Queue depths and processing rates

• Connection pool health

• Error rates and failed operations

Set up alerting for critical thresholds that could impact MDSS performance.

Backup & Disaster Recovery

Establish regular backup procedures and test recovery processes for all external services. For managed services, configure automated backups with appropriate retention policies.

Performance Optimization

• Configure connection pooling based on your concurrent load requirements

• Set appropriate resource limits and file descriptors

• Monitor and tune service-specific performance parameters

• Plan capacity based on expected MDSS throughput

Documentation References

For detailed configuration guidance, refer to the official production documentation:

• PostgreSQL Documentation

• RabbitMQ Production Checklist

• Redis Security Guidelines

For managed cloud services, consult your cloud provider's best practices documentation for the specific services you're using.