Container-Based Setup

Prerequisite

Before running the setup, please check Recommended System Configuration to install all required dependencies of MetaDefender Distributed Cluster (MDDC).

Setup order requirement

Please follow installation steps in order to complete the system setup properly:

Order no.

Service

Notes

1

Redis, RabbitMQ, PostgreSQL, Identity Service and File Storage

  • Could be setup in parallel in any order among them.

  • Make sure they are all fully functional and accessible before proceeding to the next setup order #2.

2

Control Center

  • Ensure it's able to connect to those services in #1

  • Make sure it is fully functional and accessible.

3

Worker for API Gateway and Worker for MetaDefender Core

  • Could be setup in parallel in any order among them.

  • Ensure they're able to connect to Control Center

  • Make sure they are all fully functional and accessible.

Image name and version

All the images can be found at OPSWAT Docker Hub with the following information:

Note: version is the currently release version.

MetaDefender Distributed Cluster Identity Service

opswat/metadefender-distributed-cluster:identity-service-<version>-debian-12

MetaDefender Distributed Cluster File Storage

opswat/metadefender-distributed-cluster:file-storage-<version>-debian-12

MetaDefender Distributed Cluster Control Center

opswat/metadefender-distributed-cluster:control-center-<version>-debian-12

MetaDefender Distributed Cluster Worker for API Gateway

opswat/metadefender-distributed-cluster:worker-api-gateway-<version>-debian-12

MetaDefender Distributed Cluster Worker for MetaDefender Core

opswat/metadefender-distributed-cluster:worker-core-<version>-debian-12

Environment variables

1. MDDC Identity Service

Environment Variable

Necessity

Description

MDDC_IDENTITY_SERVICE_DB_HOST

Required

Provide the database host for Identity Service

MDDC_IDENTITY_SERVICE_DB_PORT

Optional

Provide the database port for Identity Service Default: 5432

MDDC_IDENTITY_SERVICE_DB_USER

Required

Provide the database user for Identity Service

MDDC_IDENTITY_SERVICE_DB_PASSWORD

Required

Provide the database password for Identity Service

MDDC_USER

Required

Define the information to initiate the administrator account. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_PASSWORD

Required

Define the information to initiate the administrator account. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_EMAIL

Required

Define the information to initiate the administrator account. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_APIKEY

Optional

Define the information to initiate the administrator account. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_IDENTITY_SERVICE_CONNECTION_KEY

Required

Define the connection key in order to register to Control Center. Must be 4 to 64 characters long, using only letters and digits (0–9, a–z, A–Z).

MDDC_IDENTITY_SERVICE_PORT

Optional

Define the expose port for Identity Service Default: 8891

LOG_LEVEL

Optional

Define the log level. Default value: info

Accepted values: info/debug/error/warning

Start Identity Service container with docker run:

docker run -d --name mddc-identity-service \ -e MDDC_IDENTITY_SERVICE_DB_HOST=postgres-is \ -e MDDC_IDENTITY_SERVICE_DB_USER=admin \ -e MDDC_IDENTITY_SERVICE_DB_PASSWORD=admin \ -e MDDC_IDENTITY_SERVICE_CONNECTION_KEY=1234abcd \ -e MDDC_USER=admin \ -e MDDC_PASSWORD=admin \ -e MDDC_EMAIL=admin@local \ -p 8891:8891 opswat/metadefender-distributed-cluster:identity-service-2.0.0-debian-12

2. MDDC File Storage

Environment Variable

Necessity

Description

MDDC_FILE_STORAGE_CONNECTION_KEY

Required

Define the connection key in order to register to Control Center. Must be 4 to 64 characters long, using only letters and digits (0–9, a–z, A–Z).

MDDC_FILE_STORAGE_PORT

Optional

Define the expose port for File Storage Default: 8890

LOG_LEVEL

Optional

Define the log level. Default value: info

Accepted values: info/debug/error/warning

Note: Persistent storage is located at /opt/opswat/mddc-file-storage. If end-users require data to be retained across container lifecycles, they must mount a volume to this path with 777 permissions to ensure full read/write access for all processes.

Start File Storage container with docker run

docker run -d --name mddc-file-storage \ -e MDDC_FILE_STORAGE_CONNECTION_KEY=1234abcd \ -e MDDC_FILE_STORAGE_PORT=8890 \ -p 8890:8890 opswat/metadefender-distributed-cluster:file-storage-2.0.0-debian-12

3. MDDC Control Center

Environment Variable

Necessity

Description

MDDC_LAKE_DB_HOST

Optional

Provide the database host for Data Lake. In case that the end-user does not have the Data Lake, it's required to provide this variable to automate the database preparation

MDDC_LAKE_DB_PORT

Optional

Provide the database port for Data Lake Default: 5432

MDDC_LAKE_DB_USER

Optional

Provide the database user for Data Lake. In case that the end-user does not have the Data Lake, it's required to provide this variable to automate the database preparation

MDDC_LAKE_DB_PASSWORD

Optional

Provide the database password for Data Lake. In case that the end-user does not have the Data Lake, it's required to provide this variable to automate the database preparation

MDDC_WAREHOUSE_DB_HOST

Optional

Provide the database host for Data Warehouse. In case that the end-user does not have the Data Warehouse, it's required to provide this variable to automate the database preparation

MDDC_WAREHOUSE_DB_PORT

Optional

Provide the database port for Data Warehouse Default: 5432

MDDC_WAREHOUSE_DB_USER

Optional

Provide the database user for Data Warehouse. In case that the end-user does not have the Data Warehouse, it's required to provide this variable to automate the database preparation

MDDC_WAREHOUSE_DB_PASSWORD

Optional

Provide the database password for Data Warehouse. In case that the end-user does not have the Data Warehouse, it's required to provide this variable to automate the database preparation

MDDC_CACHE_HOST

Optional

Provide the caching host (Redis).

MDDC_CACHE_PORT

Optional

Provide the caching port (Redis)

MDDC_BROKER_HOST

Optional

Provide the broker host (RabbitMQ)

MDDC_BROKER_PORT

Optional

Provide the broker port (RabbitMQ)

MDDC_BROKER_USER

Optional

Provide the broker username (RabbitMQ)

MDDC_BROKER_PASSWORD

Optional

Provide the broker password (RabbitMQ)

MDDC_CONTROL_CENTER_DB_HOST

Required

Provide the database host for Control Center

MDDC_CONTROL_CENTER_DB_PORT

Optional

Provide the database port for Control Center Default: 5432

MDDC_CONTROL_CENTER_DB_USER

Required

Provide the database username for Control Center

MDDC_CONTROL_CENTER_DB_PASSWORD

Required

Provide the database password for Control Center

MDDC_USER

Required

Provide the administrator account that is defined in Identity Service. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_PASSWORD

Required

Provide the administrator account that is defined in Identity Service. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_APIKEY

Optional

Provide the administrator account that is defined in Identity Service. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_IDENTITY_SERVICE_HOST

Required

Provide the Identity Service host in order to add it to Control Center

MDDC_IDENTITY_SERVICE_PORT

Optional

Provide the Identity Service port in order to add it to Control Center Default: 8891

MDDC_IDENTITY_SERVICE_CONNECTION_KEY

Required

Provide the Identity Service connection key in order to add it to Control Center. Must be 4 to 64 characters long, using only letters and digits (0–9, a–z, A–Z).

MDDC_CONTROL_CENTER_ENCRYPTION_KEY

Required

Define the encryption key for communication between Control Center and the services. Must be 32 characters long and contain only lowercase letters (a–z) and digits (0–9)

MDDC_FILE_STORAGE_HOST

Required

Provide the File Storage host in order to add it to Control Center

MDDC_FILE_STORAGE_PORT

Optional

Provide the File Storage port in order to add it to Control Center Default: 8890

MDDC_FILE_STORAGE_CONNECTION_KEY

Required

Provide the File Storage connection key in order to add it to Control Center. Must be 4 to 64 characters long, using only letters and digits (0–9, a–z, A–Z).

MDDC_CERT_PATH

Optional

Provide the directory path that contains the certificate and private key in order to enable https Note: when provide this variable, it's supposed to mount this path to /certs/ as volume For example: --volume /your-path:/certs

LOG_LEVEL

Optional

Define the log level. Default value: info

Accepted values: info/debug/error/warning

Start Control Center container with Docker run

docker run -d --name mddc-control-center \ -e MDDC_CONTROL_CENTER_DB_HOST=postges-cc \ -e MDDC_CONTROL_CENTER_DB_USER=admin \ -e MDDC_CONTROL_CENTER_DB_PASSWORD=admin \ -e MDDC_IDENTITY_SERVICE_HOST=mddc-identity-service \ -e MDDC_USER=admin \ -e MDDC_PASSWORD=admin \ -e MDDC_IDENTITY_SERVICE_CONNECTION_KEY=1234abcd \ -e MDDC_CONTROL_CENTER_ENCRYPTION_KEY=12345678123456781234567812345678 \ -e MDDC_FILE_STORAGE_HOST=mddc-file-storage \ -e MDDC_FILE_STORAGE_CONNECTION_KEY=1234abcd \ -e MDDC_CERT_PATH=/certs \ -v /new-certificates:/certs -p 8892:8892 opswat/metadefender-distributed-cluster:control-center-2.0.0-debian-12

4. MDDC Worker for API Gateway

Environment Variable

Necessity

Description

MDDC_WORKER_CONNECTION_KEY

Required

Define the connection key in order to register to Control Center. Must be 4 to 64 characters long, using only letters and digits (0–9, a–z, A–Z).

MDDC_WORKER_PORT

Optional

Define the expose worker's port. Default: 8893

MDDC_WORKER_HOST

Optional

Define the worker's host address. If it's not specified, it will get the container's internal IP address

MDDC_CONTROL_CENTER_HOST

Required

Provide the Control Center's host address

MDDC_CONTROL_CENTER_PORT

Optional

Provide the Control Center's port Default: 8892

MDDC_USER

Required

Provide the administrator account that is defined in Identity Service. It can be optional if the end-user provides the MDDC_APIKEY. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_PASSWORD

Required

Provide the administrator account that is defined in Identity Service. It can be optional if the end-user provides the MDDC_APIKEY. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_APIKEY

Optional

Provide the administrator account that is defined in Identity Service. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_API_GATEWAY_PORT

Optional

Define the expose port to scan files via API Gateway Default: 8899

LOG_LEVEL

Optional

Define the log level. Default value: info

Accepted values: info/debug/error/warning

Start Worker for API Gateway container with Docker run

docker run -d --name mddc-worker-api-gateway \ -e MDDC_WORKER_CONNECTION_KEY=1234abcd \ -e MDDC_WORKER_HOST=mddc-core-api-gateway \ -e MDDC_CONTROL_CENTER_HOST=mddc-control-center \ -e MDDC_USER=admin \ -e MDDC_PASSWORD=admin \ -e MDDC_API_GATEWAY_PORT=8899 \ -p 8893:8893 -p 8899:8899 opswat/metadefender-distributed-cluster:worker-api-gateway-2.0.0-debian-12

5. MDDC Worker for Core

Environment Variable

Necessity

Description

MDDC_WORKER_CONNECTION_KEY

Required

Define the connection key in order to register to Control Center. Must be 4 to 64 characters long, using only letters and digits (0–9, a–z, A–Z).

MDDC_WORKER_PORT

Optional

Define the expose worker's port. Default: 8893

MDDC_WORKER_HOST

Optional

Define the worker's host address. If it's not specified, it will get the container's internal IP address

MDDC_CONTROL_CENTER_HOST

Required

Provide the Control Center's host address

MDDC_CONTROL_CENTER_PORT

Optional

Provide the Control Center's port. Default: 8892

MDDC_USER

Required

Provide the administrator account that is defined in Identity Service. It can be optional if the end-user provides the MDDC_APIKEY. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_PASSWORD

Required

Provide the administrator account that is defined in Identity Service. It can be optional if the end-user provides the MDDC_APIKEY. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

MDDC_APIKEY

Optional

Provide the administrator account that is defined in Identity Service. This account is to automatically do the following tasks:

  • Add Redis to Control Center if specified

  • Add RabbitMQ to Control Center if specified

  • Add Data Lake to Control Center if specified

  • Add Data Warehouse to Control Center if specified

  • Add File Storage to Control Center if specified

  • Add Worker to Control Center, upload API Gateway installer to Control Center, and deploy API Gateway to Worker

  • Add Worker to Control Center, upload MetaDefender Core installer to Control Center, and deploy MetaDefender Core to Worker

LOG_LEVEL

Optional

Define the log level. Default value: info

Accepted values: info/debug/error/warning

MDDC_LICENSE_KEY

Optional

Provide the license key to activate MetaDefender Core

MDDC_LICENSE_DESCRIPTION

Optional

Define description of the license key

Start Worker for Core container with Docker run:

docker run -d --name mddc-worker-core \ -e MDDC_WORKER_CONNECTION_KEY=1234abcd \ -e MDDC_WORKER_HOST=mddc-core-core \ -e MDDC_CONTROL_CENTER_HOST=mddc-control-center \ -e MDDC_USER=admin \ -e MDDC_PASSWORD=admin \ -p 8893:8893 opswat/metadefender-distributed-cluster:worker-core-2.0.0-debian-12

Start MetaDefender Distributed Cluster with Docker Compose

  1. Create a local file named docker-compose.yaml and copy the following content to this file:

services: redis: image: redis:7.0.5 container_name: redis ports: - "6379:6379" networks: - mddc rabbitmq: image: rabbitmq:3.13.0 container_name: rabbitmq restart: always healthcheck: test: ["CMD", "rabbitmq-diagnostics", "-q","ping"] interval: 10s timeout: 10s retries: 30 start_period: 20s environment: - RABBITMQ_DEFAULT_USER=admin - RABBITMQ_DEFAULT_PASS=admin ports: - "5672:5672" - "15672:15672" networks: - mddc postgres: image: postgres:14.17 container_name: postgres ports: - 5432:5432 networks: - mddc environment: - POSTGRES_USER=admin - POSTGRES_PASSWORD=admin healthcheck: test: [ "CMD", "pg_isready", "-U", "admin", "-d", "postgres" ] interval: 10s timeout: 10s retries: 30 identity-service: env_file: - .env.example image: opswat/metadefender-distributed-cluster:identity-service-2.0.0-debian-12 container_name: identity-service ports: - 8891:8891 networks: - mddc deploy: restart_policy: condition: on-failure depends_on: postgres: condition: service_healthy restart: true file-storage: env_file: - .env.example image: opswat/metadefender-distributed-cluster:file-storage-2.0.0-debian-12 container_name: file-storage ports: - 8890:8890 networks: - mddc deploy: restart_policy: condition: on-failure depends_on: postgres: condition: service_healthy restart: true control-center: env_file: - .env.example image: opswat/metadefender-distributed-cluster:control-center-2.0.0-debian-12 container_name: control-center ports: - 8892:8892 networks: - mddc deploy: restart_policy: condition: on-failure healthcheck: test: ["CMD", "true"] interval: 60s start_period: 90s start_interval: 60s depends_on: - identity-service - redis - rabbitmq - file-storage worker-api-gateway: env_file: - .env.example image: opswat/metadefender-distributed-cluster:worker-api-gateway-2.0.0-debian-12 container_name: worker-api-gateway ports: - "8893" - 7777:7777 networks: - mddc deploy: restart_policy: condition: on-failure healthcheck: test: ["CMD", "true"] interval: 5s timeout: 2s start_period: 30s depends_on: control-center: condition: service_healthy worker-core: env_file: - .env.example image: opswat/metadefender-distributed-cluster:worker-core-2.0.0-debian-12 container_name: worker-core ports: - "8008" networks: - mddc deploy: restart_policy: condition: on-failure depends_on: control-center: condition: service_healthy worker-api-gateway: condition: service_healthy networks: mddc: driver: bridge ipam: config: - subnet: 10.0.0.0/24 gateway: 10.0.0.1
  1. Prepare an environment variable file named .env.example and provide with your own values

  2. Run the application with the command:

docker compose up -d

Known limitation

  • When the host experiences resource limitations or degraded performance, some containers may fail to start properly. In such cases, restarting the container is recommended to restore normal operation.