Inspecting a deployment
Pod status
kubectl get pods
kubectl get pods -l component=worker # all worker instances
kubectl describe pod <pod> # events, probe failures, scheduling problems
The Events section at the end of describe is usually where the answer is: image pull failures, unschedulable resource requests, and probe failures all appear there.
Logs
# Follow a service
kubectl logs -f deploy/control-center
kubectl logs -f deploy/identity-service
# A specific StatefulSet replica
kubectl logs -f file-storage-0
kubectl logs -f ometascan-0
# The Control Center's installer initContainer
kubectl logs deploy/control-center -c installers
# The previous container, after a crash loop
kubectl logs ometascan-0 --previous
# A pre-upgrade Job
kubectl logs job/ometascan-pre-upgrade
Run a health check by hand
Every service container carries the same health check script that its probes use. Running it directly shows you why a probe is failing:
kubectl exec ometascan-0 -- sh -c '${WORK_DIR}/healthcheck.sh readiness'
kubectl exec ometascan-0 -- sh -c '${WORK_DIR}/healthcheck.sh liveness'
kubectl exec deploy/control-center -- sh -c '${WORK_DIR}/healthcheck.sh readiness'
Readiness detail
The MetaDefender Cluster Control Center reports per-component readiness:
kubectl port-forward deploy/control-center 8892:8892
curl -s 'http://127.0.0.1:8892/readyz?verbose=true' | jq
This names the component that is not ready — database, cache, broker or MetaDefender Cluster Identity Service — which is faster than reading the log.
Configuration actually in effect
kubectl get configmap mdcluster-config -o yaml
kubectl get secret mdcluster-secrets -o jsonpath='{.data}' | jq 'keys' # key names only
helm get values services
helm get values instances
helm get values shows what the release really holds, which is not always what your values file says — particularly after an upgrade with --reuse-values.
Common problems
Symptom | Check |
|---|
Pods stuck in ImagePullBackOff | DOCKER_REPO and MDCLS_VERSION resolve to a tag that exists, and the pull secret named in imagePullSecrets exists in this namespace.
|
MetaDefender Cluster Control Center never becomes ready | Work through its pre-start dependencies in order, all visible in the container log: the MetaDefender Cluster Control Center database port → database creation and migration (logged, but its failure does not stop the container) → the MetaDefender Cluster Identity Service port → every MetaDefender Cluster File Storage endpoint. After the service starts, GET /readyz?verbose=true names the component still failing — datalake and warehouse problems surface here, not in the pre-start checks. Verify CONTROL_CENTER_DB_HOST, DATALAKE_SERVICES, WAREHOUSE_SERVICES, the credentials, and that the role can create databases. Raise env.GLOBAL_WAIT_TIMEOUT on slow clusters — but note it also lengthens MetaDefender Cluster Worker shutdown, which shares the same bound. |
MetaDefender Cluster Control Center restarts, and one MetaDefender Cluster File Storage pod is missing | The MetaDefender Cluster Control Center waits for every endpoint in its list before it starts. One MetaDefender Cluster File Storage replica that never comes up keeps the MetaDefender Cluster Control Center in a restart loop rather than merely reducing capacity. |
MetaDefender Cluster Control Center crash-loops immediately | CONTROL_CENTER_ENCRYPTION_KEY or ADMIN_APIKEY is empty. The container refuses to start with either unset.
|
MetaDefender Cluster Control Center CrashLoopBackOff with Failed to encrypt configuration | CONTROL_CENTER_ENCRYPTION_KEY is not exactly 32 characters. The length is never validated, so the container launches, then fails to encrypt every service registration — no service registers, /readyz never returns 200, and the container is killed after GLOBAL_WAIT_TIMEOUT. kubectl logs deploy/control-center | grep -i encrypt.
|
MetaDefender Cluster Workers CrashLoopBackOff right after install | MetaDefender Cluster Workers depend on the MetaDefender Cluster Control Center. Confirm mdcluster-config and mdcluster-secrets exist — installing md-cluster-instances without md-cluster-services leaves them missing — and that the MetaDefender Cluster Control Center is ready. |
MetaDefender Cluster Worker pod Running but never Ready | Readiness covers the agent and the deployed instance: the agent's own /readyz on 8893, an installed instance directory on disk, then the instance's /readyz. Run the check by hand to see which of the three fails. For ometascan, engine initialisation legitimately takes several minutes on first start. |
MetaDefender Cluster Worker instance never deploys | The MetaDefender Cluster Control Center must hold an installer of this instance type and list this MetaDefender Cluster Worker as eligible to deploy it (a version-compatibility check). The MetaDefender Cluster Worker retries the installer search only three times, five seconds apart, then unregisters itself and restarts — so a slow installer upload appears as a MetaDefender Cluster Worker crash loop, not a wait. Check the installers initContainer log, confirm the MetaDefender Cluster Control Center finished its startup sequence, and confirm the installer version supports the MetaDefender Cluster Worker version. |
MetaDefender Cluster Worker pod is Ready but scans fail as unlicensed | Liveness only fails on failed and disconnected, so an unlicensed worker is never restarted or flagged. kubectl logs <pod> | grep -i licen and check activation slots. |
Cannot reach the MetaDefender Cluster Control Center console | The NodePort is assigned by Kubernetes unless you set control-center.service.nodePort. Run kubectl get svc control-center to see the actual port, use kubectl port-forward, or switch the Service to LoadBalancer. |
MetaDefender Cluster File Storage data lost on restart | file-storage.persistence.enabled must be true. The default emptyDir is deleted with the pod.
|
MetaDefender Cluster Control Center CrashLoopBackOff roughly every GLOBAL_WAIT_TIMEOUT seconds | env.FILE_STORAGE_MAX_REPLICA exceeds the number of reachable MetaDefender Cluster File Storage instances. Log line: Max replica must be less than the number of instances. It must be ≤ file-storage.replicas.
|
MetaDefender Cluster Control Center dies seconds after becoming ready | env.FILE_STORAGE_MIN_REPLICA is greater than env.FILE_STORAGE_MAX_REPLICA. Rejected immediately with no retry. Log line: Min replica must be less than or equal to max replica.
|
MetaDefender Cluster File Storage scaled, but new capacity not used | Scaling must go through helm upgrade, not kubectl scale — the MetaDefender Cluster Control Center's endpoint list is built at render time. All new pods must be Running and registered before the capacity activates. See Scaling. |
PVC stuck Pending | No default StorageClass, or the named class cannot provision the requested size. kubectl get storageclass and kubectl describe pvc <name>. |
Pod stuck Pending with no PVC involved | No node satisfies the resource requests. ephemeral-storage is the usual culprit. kubectl describe pod <pod>. |
MetaDefender Cluster Worker pod evicted mid-scan | It exceeded its ephemeral-storage limit. Raise it, or set one if absent. See Resource requirements. |
Redis READONLY errors | MetaDefender Cluster cannot use a Sentinel endpoint, so writes land on replicas. Front Redis with a single writable endpoint — the redis-ha chart's own HAProxy (haproxy.enabled=true in that chart's values, not in override-values.yaml), or a managed Redis. See Redis. |
Authentication failures against external infrastructure | Every *_DB_PASSWORD must equal the database role's password, and RABBITMQ_PASSWORD must equal the broker chart's password. See High availability |
Instances unlicensed after a rollout or scale-out | Not enough activation slots, or activations stranded by MetaDefender Cluster Workers that did not shut down gracefully. See Licensing in Kubernetes. |
helm upgrade fails on a pre-upgrade Job
| The Jobs have backoffLimit: 0, so one failure fails the upgrade. Read kubectl logs job/<instance>-pre-upgrade before retrying. |
MetaDefender Cluster Worker killed before unregistering | terminationGracePeriodSeconds expired during shutdown. It must exceed isolate.timeout plus GLOBAL_WAIT_TIMEOUT, since the undeploy step is bounded separately. See Graceful shutdown.
|
Diagnosing specific areas
Database connectivity
The MetaDefender Cluster Control Center creates four databases on first start. Confirm they exist:
kubectl exec deploy/postgres -- psql -U <db-user> -d postgres -c '\l'
Expect md_cluster_control_center, md_cluster_datalake, md_cluster_datawarehouse and md_cluster_identity_service.
If they are missing, the database role most likely cannot create databases. With the bundled PostgreSQL this happens when the four credential pairs in secrets: disagree — the bundled instance takes its superuser credentials from CONTROL_CENTER_DB_USER and CONTROL_CENTER_DB_PASSWORD, so the MetaDefender Cluster Identity, datalake and warehouse users must match. With an external database, the role needs createdb.
Check connection exhaustion when running many replicas:
kubectl exec deploy/postgres -- psql -U <db-user> -d postgres -c \
'SELECT count(*), setting FROM pg_stat_activity, pg_settings WHERE name = $$max_connections$$ GROUP BY setting;'
Raise postgres.maxConnections if you are near the limit.
Service discovery
Verify a name resolves from inside a pod:
kubectl exec deploy/control-center -- getent hosts identity-service
kubectl exec deploy/control-center -- getent hosts file-storage-0.file-storage
If components live in different namespaces, endpoints must be fully qualified as <service>.<namespace>.svc.cluster.local.
kubectl get pvc
kubectl exec file-storage-0 -- df -h /app/storage
kubectl exec deploy/postgres -- df -h /var/lib/postgresql/data
Permission errors writing to /app/storage normally mean the volume does not honour the pod's fsGroup. This is the expected failure mode with NFS, which is not supported — see Storage configuration.
Collecting information for support
# Everything in the namespace
kubectl get all -o wide > cluster-state.txt
kubectl describe pods > pod-details.txt
# Effective configuration (Secret values are not included)
kubectl get configmap mdcluster-config -o yaml > config.yaml
helm get values services > values-services.yaml
helm get values instances > values-instances.yaml
# Logs
for p in $(kubectl get pods -o name); do
kubectl logs "$p" --all-containers --tail=5000 > "logs-${p#pod/}.txt"
done
Include your MetaDefender Cluster version (MDCLS_VERSION), your Kubernetes version (kubectl version), and whether you use the bundled or external infrastructure.
Review the files before sharing them — logs and ConfigMap contents may include hostnames and account names from your environment.
