High availability

The bundled postgres, redis and rabbitmq are single-pod deployments. They exist so that a fresh install works out of the box and matches the Docker Compose stack — not for production. Each is a single point of failure whose loss stops the cluster.

A highly available deployment has three parts:

Run each stateful dependency as an external clustered deployment.
Disable the bundled single-pod infrastructure.
Point MetaDefender Cluster at the external endpoints and run multiple replicas of MetaDefender Cluster File Storage and the MetaDefender Cluster Workers.

Note what cannot be made highly available: MetaDefender Cluster Control Center and MetaDefender Cluster Identity Service are singletons, fixed at one replica by the chart. Redundancy for them comes from Kubernetes restarting the pod, not from running two.

The two behave differently during an upgrade. The MetaDefender Cluster Control Center is pinned to a recreate-style rollout, so its old pod is terminated before the replacement starts — expect a short orchestration gap. MetaDefender Cluster Identity Service uses the default rolling strategy, so its replacement starts first and there is no gap, at the cost of two MetaDefender Cluster Identity Service pods overlapping briefly.

MetaDefender Cluster Workers already running continue to serve traffic through a MetaDefender Cluster Control Center outage.

Choosing an approach

In production, prefer a managed service from your cloud provider. Azure, AWS and Google Cloud all offer PostgreSQL-compatible databases with built-in high availability, backups and dynamic scaling — considerably less to operate than a database cluster you run yourself.

If the dependency must live inside the Kubernetes cluster, the options below are tested:

Dependency	On-cluster solution	Endpoint MetaDefender Cluster uses
PostgreSQL	CloudNativePG operator, 3 instances	`md-postgres-ha-rw:5432` (primary, read-write)
Redis	dandydeveloper/redis-ha (Redis + Sentinel), 3 nodes, HAProxy enabled	`md-redis-ha-haproxy:6379` — a single writable endpoint
RabbitMQ	RabbitMQ Cluster Operator, 3 nodes	The 3 broker pods via the headless Service

Namespace and DNS. The Service names below assume the HA components run in the same namespace as the md-cluster-services and md-cluster-instances releases. If they live elsewhere, use the fully qualified form <service>.<namespace>.svc.cluster.local.

RabbitMQ

For production, use the official RabbitMQ Cluster Operator.

The Bitnami community chart also works and is quicker to stand up, but note that it now ships images from the bitnamilegacy repository and is best-effort and unsupported.

Create bitnami_rabbitmq.yaml:

replicaCount: 3 persistence: enabled: true size: 50Gi auth: username: '<rabbitmq-user>' password: '<rabbitmq-password>' erlangCookie: '<erlang-cookie>' image: registry: docker.io repository: bitnamilegacy/rabbitmq global: security: allowInsecureImages: true # resources: # requests: { cpu: "4", memory: "8Gi" } # limits: { cpu: "4", memory: "8Gi" }

helm repo add bitnami https://charts.bitnami.com/bitnami helm upgrade --install md-rabbitmq-ha bitnami/rabbitmq \ --version 16.0.14 -f ./bitnami_rabbitmq.yaml

Pin the chart version. The values above set image.repository but not image.tag, so the tag floats with the chart. bitnamilegacy is a frozen mirror, so a later chart version will ask for a tag that was never mirrored there and the pods fail with ImagePullBackOff and no obvious cause. Pin the chart version, or pin image.tag too.

This creates pods md-rabbitmq-ha-0, -1 and -2, reachable through the headless Service md-rabbitmq-ha-headless.

Redis

Create dandy_redis.yaml:

persistentVolume: enabled: false redis: disableCommands: null # keep FLUSHDB/FLUSHALL enabled — the scan engines use them haproxy: enabled: true # required: gives one writable endpoint at md-redis-ha-haproxy:6379 # resources: # requests: { cpu: "2", memory: "8Gi" } # limits: { cpu: "2", memory: "8Gi" } # sentinel: # resources: # requests: { cpu: "100m", memory: "256Mi" } # limits: { cpu: "200m", memory: "512Mi" }

helm repo add dandydev https://dandydeveloper.github.io/charts helm upgrade --install md-redis-ha dandydev/redis-ha -f ./dandy_redis.yaml

This creates pods md-redis-ha-server-0, -1 and -2, reachable through the md-redis-ha Service.

disableCommands: null is required. The chart disables FLUSHDB and FLUSHALL by default, and the deployed MetaDefender Core instances use them.

Give the deployment a dedicated Redis. Because MetaDefender Core issues FLUSHDB, pointing REDIS_SERVICES at an instance shared with another application risks discarding that application's data.

MetaDefender Cluster is not Sentinel-aware. It connects to Redis using a plain host:port list, so it has no way to discover which node is currently the primary. Writes that land on a replica fail with READONLY errors.

Front the cluster with a single writable endpoint instead. Either enable the chart's HAProxy (haproxy.enabled=true, giving one endpoint at md-redis-ha-haproxy:6379), or use a managed Redis that presents a single writable address.

PostgreSQL

CloudNativePG runs PostgreSQL as an operator-managed cluster with automatic failover.

Create the credentials Secret

cloudpg_secret.yaml:

apiVersion: v1 kind: Secret metadata: name: md-postgres-ha-secret type: kubernetes.io/basic-auth stringData: username: '<db-user>' # must match the role name below password: '<db-password>' # must match every *_DB_PASSWORD / *_PASSWORD value

Define the cluster

cloudpg_cluster.yaml:

apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: md-postgres-ha spec: instances: 3 storage: size: 100Gi # storageClass: "" # resources: # requests: { cpu: "4", memory: "8Gi" } # limits: { cpu: "4", memory: "8Gi" } managed: roles: - name: '<db-user>' passwordSecret: name: md-postgres-ha-secret login: true createdb: true # required: the Control Center creates the four cluster databases superuser: true postgresql: parameters: log_min_duration_statement: "300" # log queries slower than 300ms log_line_prefix: "%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h " log_checkpoints: "off" log_connections: "off" log_disconnections: "off" log_lock_waits: "on"

The role needs createdb and superuser because the MetaDefender Cluster Control Center's database initialisation step creates the four cluster databases on first start: md_cluster_control_center, md_cluster_datalake, md_cluster_datawarehouse and md_cluster_identity_service.

Deploy

Install the operator with cluster-admin privileges — the manifest creates cluster-scoped resources:

kubectl apply --server-side -f \ https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml

Then apply the Secret and the cluster:

kubectl apply -f cloudpg_secret.yaml kubectl apply -f cloudpg_cluster.yaml

CloudNativePG exposes three Services: md-postgres-ha-rw (the current primary, read-write), md-postgres-ha-ro (replicas) and md-postgres-ha-r (any instance). MetaDefender Cluster must use -rw. The endpoint follows the primary automatically on failover.

MetaDefender Cluster File Storage

MetaDefender Cluster File Storage is part of the md-cluster-services chart rather than an external dependency, but it holds scanned files and their metadata, so it needs its own treatment.

How it works

Each pod — file-storage-0, file-storage-1, and so on — gets its own ReadWriteOnce PVC mounted at /app/storage, plus a stable DNS name from the headless Service.

The pods do not register themselves. The chart builds the complete endpoint list at render time from file-storage.replicas and hands it to the MetaDefender Cluster Control Center, which connects to each address. Each file is then written to several instances, the number of copies being set by FILE_STORAGE_MIN_REPLICA and FILE_STORAGE_MAX_REPLICA.

Because the list is fixed at render time, changing the replica count requires helm upgrade rather than kubectl scale.

FILE_STORAGE_MIN_REPLICA is a redundancy floor with teeth. Setting it to 2 does not just mean "keep two copies" — if fewer than two MetaDefender Cluster File Storage instances are healthy, the MetaDefender Cluster Control Center reports MetaDefender Cluster File Storage unhealthy and refuses writes and lookups, and its own readiness check fails. Losing one MetaDefender Cluster File Storage pod then takes the MetaDefender Cluster Control Center down with it. Size the replica count so the floor stays satisfied while a pod is being replaced.

Configuration

env: FILE_STORAGE_MIN_REPLICA: 2 FILE_STORAGE_MAX_REPLICA: 2 file-storage: replicas: 3 persistence: enabled: true # one PVC per replica; false means ephemeral emptyDir size: 100Gi # per replica

FILE_STORAGE_MAX_REPLICA must be less than or equal to file-storage.replicas, and FILE_STORAGE_MIN_REPLICA less than or equal to FILE_STORAGE_MAX_REPLICA.

Requirements

Persistence must be enabled. With persistence.enabled: false each replica uses an ephemeral emptyDir and loses its files on restart, which defeats the point of running several.

Each replica needs its own ReadWriteOnce volume. Do not point the replicas at a single shared ReadWriteMany volume — each instance owns its own metadata database and expects exclusive access.

Spread the replicas across nodes. By default nothing prevents all MetaDefender Cluster File Storage pods from landing on the same node, which would make node loss a total outage. Add anti-affinity:

file-storage: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - topologyKey: kubernetes.io/hostname labelSelector: matchLabels: app: file-storage

Use requiredDuringScheduling only if you have at least as many eligible nodes as replicas; otherwise the surplus pods stay Pending. Use preferredDuringScheduling for a best-effort spread. For zone-level fault tolerance, combine this with a zone-aware StorageClass — a pod cannot move to another zone if its volume is pinned to the first one.

MetaDefender Cluster File Storage runs as UID and GID 1000; the chart already sets fsGroup: 1000 so mounted volumes are writable. See Storage configuration.

On an existing deployment, scale all MetaDefender Cluster Worker pods to zero before changing the MetaDefender Cluster File Storage replica count. The MetaDefender Cluster Control Center rejects the new copy settings while any MetaDefender Cluster Worker still has an instance deployed — and it treats that rejection as already-applied, so your settings are silently ignored rather than reported as an error. The full procedure is in Scaling. This does not apply to a fresh install, where there are no MetaDefender Cluster Workers yet - see MetaDefender Cluster File Storage.

Point MetaDefender Cluster at the external infrastructure

Add the following to your values file. Passwords must match the RabbitMQ chart auth and the CloudNativePG role Secret exactly — a mismatch produces authentication failures at startup rather than a clear configuration error.

# Disable the bundled single-pod infrastructure postgres: { enabled: false } redis: { enabled: false } rabbitmq: { enabled: false } secrets: CONTROL_CENTER_ENCRYPTION_KEY: '<32-character-key>' ADMIN_APIKEY: '<control-center-api-key>' # LICENSE_KEY: '<license-key>' # PostgreSQL — all four pairs must match the CloudNativePG role CONTROL_CENTER_DB_USER: '<db-user>' CONTROL_CENTER_DB_PASSWORD: '<db-password>' IDENTITY_DB_USER: '<db-user>' IDENTITY_DB_PASSWORD: '<db-password>' DATALAKE_USER: '<db-user>' DATALAKE_PASSWORD: '<db-password>' WAREHOUSE_USER: '<db-user>' WAREHOUSE_PASSWORD: '<db-password>' # RabbitMQ — must match bitnami_rabbitmq.yaml auth RABBITMQ_USER: '<rabbitmq-user>' RABBITMQ_PASSWORD: '<rabbitmq-password>' # REDIS_USER / REDIS_PASSWORD — only if your Redis requires auth env: # PostgreSQL: always the read-write endpoint CONTROL_CENTER_DB_HOST: 'md-postgres-ha-rw' CONTROL_CENTER_DB_PORT: 5432 IDENTITY_DB_HOST: 'md-postgres-ha-rw' IDENTITY_DB_PORT: 5432 DATALAKE_SERVICES: 'md-postgres-ha-rw:5432' WAREHOUSE_SERVICES: 'md-postgres-ha-rw:5432' # RabbitMQ: all three brokers RABBITMQ_SERVICES: 'md-rabbitmq-ha-0.md-rabbitmq-ha-headless:5672,md-rabbitmq-ha-1.md-rabbitmq-ha-headless:5672,md-rabbitmq-ha-2.md-rabbitmq-ha-headless:5672' # Redis: the single writable HAProxy endpoint, never the individual nodes REDIS_SERVICES: 'md-redis-ha-haproxy:6379' # File Storage: must be <= file-storage.replicas FILE_STORAGE_MIN_REPLICA: 2 FILE_STORAGE_MAX_REPLICA: 2 # Multiple replicas of the stateful cluster services file-storage: replicas: 3 persistence: { enabled: true, size: 100Gi } workers: ometascan: { replicas: 3 } api-gateway: { replicas: 2 } callback-service: { replicas: 2 }

Install as usual:

helm install services ./md-cluster-services -f ./override-values.yaml helm install instances ./md-cluster-instances -f ./override-values.yaml

Raising MetaDefender Cluster workers.ometascan.replicas consumes one license activation per replica. Confirm you have enough, plus headroom for rollouts — see Licensing in Kubernetes.

Verify

# External infrastructure is healthy kubectl get clusters.postgresql.cnpg.io md-postgres-ha # CloudNativePG: 3 instances, primary elected kubectl get pods -l app.kubernetes.io/name=rabbitmq kubectl get pods -l app=redis-ha # MetaDefender Cluster is running with the expected replicas kubectl get pods kubectl get statefulset

Then open the Services page of the MetaDefender Cluster Control Center console. All file-storage replicas should be listed and healthy. If the count is short, see Troubleshooting.

Last updated on

Was this page helpful?