Scaling

Two kinds of scaling apply to MetaDefender Cluster, and they work very differently.

MetaDefender Cluster Workers scale freely and without downtime. They register themselves with the MetaDefender Cluster Control Center on start and unregister on stop, so adding or removing replicas needs no coordination.

MetaDefender Cluster File Storage does not. The MetaDefender Cluster Control Center is given its list of MetaDefender Cluster File Storage endpoints at chart render time, so changing the replica count requires a helm upgrade — and a planned downtime window.

MetaDefender Cluster Control Center and MetaDefender Cluster Identity Service are singletons and cannot be scaled.

MetaDefender Cluster Workers

Scale each instance type independently, according to which resource is the bottleneck: ometascan for scan throughput, api-gateway for file submission capacity, callback-service for webhook delivery.

# MetaDefender Core scan engines kubectl scale statefulset ometascan --replicas=2 # API Gateway kubectl scale statefulset api-gateway --replicas=2 # Callback Service kubectl scale statefulset callback-service --replicas=2

kubectl scale changes the live StatefulSet but not your Helm release. Whether the next helm upgrade resets it depends on your values file: if you set workers.<type>.replicas, the upgrade reasserts that number; if you leave it unset — which is the chart default — the chart omits the field entirely and Kubernetes keeps your scaled count. Either way, set it in values for a change you want to be durable and reproducible:

workers: ometascan: { replicas: 5 } api-gateway: { replicas: 3 } callback-service: { replicas: 3 }

helm upgrade instances ./md-cluster-instances -f ./override-values.yaml

What to expect

Scaling out. New pods register with the MetaDefender Cluster Control Center, deploy their instance, and become ready. For ometascan this includes initialising the MetaDefender Core engines, which takes several minutes. Because MetaDefender Cluster Worker StatefulSets use podManagementPolicy: Parallel, new pods start concurrently rather than one at a time.

Scaling in. Terminating pods run the full graceful shutdown — isolate the instance so in-flight work drains, undeploy it, then delete the MetaDefender Cluster Worker's entry in the MetaDefender Cluster Control Center. This can take several minutes per pod, by design: nothing in flight is dropped. Do not force-delete the pods to speed it up; that leaves the MetaDefender Cluster Worker entry and its license activation behind.

kubectl get pods -w kubectl rollout status statefulset/ometascan

Before scaling out `ometascan`

Each ometascan replica consumes one license activation. Scaling beyond your available activations produces pods that run but cannot scan, rather than a clear error. Check your activation count first — see Licensing in Kubernetes.

The other two instance types consume no activations.

Capacity to schedule onto

A MetaDefender Cluster Worker pod with resources.requests set will stay Pending if no node can satisfy it. ephemeral-storage is the request most often overlooked, and the least obvious when it fails:

kubectl describe pod ometascan-3 | tail -20

See Resource requirements.

Autoscaling

The charts do not ship a HorizontalPodAutoscaler, and CPU-based autoscaling is a poor fit for ometascan: a pod takes minutes to become ready and minutes to drain, so an HPA reacting to short CPU spikes will churn replicas without adding useful capacity — while each churn cycle consumes and releases a license activation.

Prefer scaling on your own operational signals — queue depth, submission rate, or a schedule that matches known traffic patterns.

MetaDefender Cluster File Storage

Never scale MetaDefender Cluster File Storage in. Files are distributed across the instances, not copied to all of them, so removing a replica removes the MetaDefender Cluster Control Center's only route to every file that lived there. With the default FILE_STORAGE_MAX_REPLICA: 1 each file has exactly one copy, so scaling in loses access to a share of your stored files immediately — and the MetaDefender Cluster Control Center reports nothing wrong.

Scaling out is safe, but offline: it needs a maintenance window with all MetaDefender Cluster Worker pods stopped. Size MetaDefender Cluster File Storage at install time so you do not need either operation.

Scaling MetaDefender Cluster File Storage requires downtime.

The reason is in how the MetaDefender Cluster Control Center learns about MetaDefender Cluster File Storage instances. Each replica has a stable per-pod DNS name (file-storage-<n>.file-storage:8890), and the chart builds the complete list at render time from file-storage.replicas, passing it to the MetaDefender Cluster Control Center as an environment variable. Scaling the StatefulSet directly with kubectl scale therefore creates pods the MetaDefender Cluster Control Center does not know exist.

Procedure

Scale all MetaDefender Cluster Worker instances to zero. MetaDefender Cluster File Storage cannot be reconfigured while MetaDefender Cluster workers are running.
kubectl scale statefulset ometascan api-gateway callback-service --replicas=0 # Wait for every worker pod to finish draining
Let the graceful shutdowns finish — do not force-delete. Each MetaDefender Cluster Worker undeploys its instance and deletes its own MetaDefender Cluster Worker entry, which is what lets the MetaDefender Cluster Control Center reclaim its license activation.
MetaDefender Cluster File Storage cannot be reconfigured while MetaDefender Cluster Workers are still registered. If you skip this step, the MetaDefender Cluster Control Center logs Service modification is forbidden while workers are in use, treats it as already-applied, and starts normally — so the new copy settings are silently ignored rather than reported as an error.
Bring the new MetaDefender Cluster File Storage pods up first, with kubectl. The MetaDefender Cluster Control Center does not know about them yet, which is exactly what you want at this point — they simply start and begin listening.
kubectl scale statefulset file-storage --replicas=3 # Wait for this to finish. Provisioning a new PVC per replica is the slow part
Now apply the change through Helm. This is what rebuilds the MetaDefender Cluster Control Center's endpoint list. Set the copy counts in the same command if the new replica count changes them:
helm upgrade services ./md-cluster-services --reuse-values \ --set file-storage.replicas=3 \ --set env.FILE_STORAGE_MIN_REPLICA=2 \ --set env.FILE_STORAGE_MAX_REPLICA=3
The StatefulSet is already at the target count from step 2, so this changes only the MetaDefender Cluster Control Center's pod template. That template change rolls the MetaDefender Cluster Control Center pod, which is how it picks up the new endpoint list — the MetaDefender Cluster Control Center reads it once, at startup, and never re-reads it.
Because every endpoint is already accepting connections, the MetaDefender Cluster Control Center starts cleanly on the first attempt.
Do not scale the MetaDefender Cluster Control Center down as part of this. It is a singleton pinned to one replica by the chart — not configurable through values — so this helm upgrade would bring it straight back regardless. Getting MetaDefender Cluster File Storage up in step 2 is what makes the restart safe; holding the MetaDefender Cluster Control Center down is neither possible nor needed.
helm upgrade instances ./md-cluster-instances --reuse-values \ --set workers.ometascan.replicas=5 \ --set workers.api-gateway.replicas=3 \ --set workers.callback-service.replicas=3

Finally, confirm on the Services page of the console that every MetaDefender Cluster File Storage replica is listed and healthy.

If you run steps 2 and 3 in the other order

Doing the helm upgrade before the pods exist still converges, but noisily: the MetaDefender Cluster Control Center restarts immediately with an endpoint list naming replicas that are not up yet, and it will not finish starting until every endpoint accepts connections. It gives up after GLOBAL_WAIT_TIMEOUT seconds and is restarted by Kubernetes, so you see a CrashLoopBackOff until the new volumes finish provisioning. Nothing is damaged — but in a maintenance window it looks like a failure, which is why the order above is worth keeping.

File copy settings

Two settings control how many copies of each file are stored. They are a redundancy factor, not a limit on how many instances the MetaDefender Cluster Control Center connects to.

Setting	Meaning
`env.FILE_STORAGE_MIN_REPLICA`	Minimum copies of each file required for a write to succeed — and minimum healthy instances. Below it, MetaDefender Cluster File Storage reports unhealthy and refuses writes and lookups entirely, which also takes the MetaDefender Cluster Control Center out of readiness.
`env.FILE_STORAGE_MAX_REPLICA`	How many copies of each file to write

This must hold:

FILE_STORAGE_MIN_REPLICA ≤ FILE_STORAGE_MAX_REPLICA ≤ file-storage.replicas

Both halves are enforced by the MetaDefender Cluster Control Center, and violating either kills the MetaDefender Cluster Control Center rather than producing a warning:

FILE_STORAGE_MAX_REPLICA greater than the number of reachable instances — retried every 5 seconds for up to GLOBAL_WAIT_TIMEOUT seconds, then the MetaDefender Cluster Control Center exits. The pod enters CrashLoopBackOff, restarting roughly every GLOBAL_WAIT_TIMEOUT seconds. The log line is Max replica must be less than the number of instances.
FILE_STORAGE_MIN_REPLICA greater than FILE_STORAGE_MAX_REPLICA — rejected immediately with no retry, so the MetaDefender Cluster Control Center dies within seconds of becoming ready. The log line is Min replica must be less than or equal to max replica.

kubectl logs deploy/control-center | grep -i replica

Scaling in

Reducing file-storage.replicas follows the same procedure. Note that Kubernetes does not delete the PVCs of removed replicas: file-storage-2's volume survives, so scaling back out later reattaches it with its files intact. Delete those PVCs only when you are certain the files are no longer needed.

kubectl get pvc -l app=file-storage

Bundled infrastructure cannot be scaled

postgres, redis and rabbitmq as shipped are single-pod deployments. Raising their replica count is not supported and would corrupt data — two PostgreSQL pods cannot share one ReadWriteOnce volume. To scale them, replace them with external clustered deployments; see High availability.

Last updated on

Was this page helpful?