Endpoint Protection Exclusions for MetaDefender Storage Security running on Windows

This article lists what to exclude from an endpoint protection product (antivirus, EDR, XDR, application control) on the Windows machines that run MDSS, and why. It is vendor neutral: each item names the protection category it applies to; map that category to the matching module in your product.

MDSS is itself a file security product. It reads files that may be malicious, deletes and moves files on storage in bulk, and runs a supervised tree of processes and PowerShell scripts. An endpoint agent is built to stop exactly these behaviours. Without exclusions it slows MDSS down, breaks scans and remediations, and in the worst case deletes customer files from the protected storage before MDSS acts on them.

Section 3 is the recommended rule set: a few generic rules that cover everything MDSS does. Appendix A is the full inventory behind those rules, for products without wildcard or parent-process features, or for policies that require every exclusion to be enumerated.

1. Symptoms of interference

2. Principles

• Exclude by full path, not by file name. Every MDSS process lives under the installation folder or is a private copy of a runtime under it. Excluding the MDSS dotnet.exe by path does not exclude other .NET applications on the host.

• Use folder and process exclusions together. A folder exclusion protects the files MDSS owns. A process exclusion protects what MDSS does, and most of that happens outside the installation folder: reading files on shares, writing temp copies, bulk deletes and moves, spawning PowerShell from C:\Windows.

• Trust the process tree. "A process and its descendants" for the mdss service is the single most effective rule in this document. If your product lacks it, exclude each executable in A.3.

• Prefer process exclusions over disabling a module, and use the OPSWAT, Inc. code signature where the product allows it (A.7).

• Do not copy this list to the MetaDefender Core host. MD Core has its own requirements.

3. Recommended exclusion rules

<INSTALL_DIR> is the installation folder, by default C:\Program Files\OPSWAT\MetaDefender Storage Security. The installer lets the customer choose another drive or folder; the real path is in the registry value InstallDir under HKLM\SOFTWARE\OPSWAT\MetaDefender Storage Security.

3.1 Folders

Exclude from on-access (real-time) scanning, scheduled scanning and behavioural file monitoring:

1. <INSTALL_DIR>\, the whole tree (A.1).

2. C:\ProgramData\OPSWAT\: upgrade state files and, under mdss\smb-mounts, one directory symbolic link per mounted SMB share pointing at \\server\share. Critical: anything the agent does under smb-mounts happens on the customer's storage. Scanning there scans the whole share through the MDSS server; quarantining there deletes the customer's file. The location is set by SMBSERVICE_MOUNT_DIRECTORY; exclude the configured value if it was changed.

3. C:\ProgramData\OPSWAT-ONTAP-Connector\, plus the folder named by LogPath in its config.ini if one is set, when the ONTAP Connector runs on this host (section 4).

4. The temp folder of the account the mdss service runs as: C:\Windows\Temp (or C:\Windows\SystemTemp on current Windows Server builds) for LocalSystem, C:\Users\<account>\AppData\Local\Temp for a custom account. MDSS writes temporary copies of the files it uploads during remediation there. They can be the original, possibly malicious, file a workflow is moving to quarantine; if the agent removes them the remediation fails. If this is too broad for your policy, the process rules in 3.2 give the same result.

3.2 Processes

Exclude from on-access scanning, network file scanning, behavioural analysis, anti-ransomware, exploit and memory protection, and .NET or process instrumentation (CLR profilers, startup hooks, APPDOMAIN_MANAGER_* injection):

1. <INSTALL_DIR>\services\opswat.mdss.windows.service\opswat.mdss.windows.service.exe and all its descendants. This is the mdss Windows service. It starts and supervises every other MDSS process: the private .NET runtime (services\dotnet\dotnet.exe, 12 or more instances hosting the microservices), nginx, the Garnet cache, the SMB sidecar, and every powershell.exe, sc.exe, icacls.exe, psql.exe, nginx.exe, node.exe and msiexec.exe it launches at start-up.

2. Any executable under <INSTALL_DIR>\services\ (<INSTALL_DIR>\services\**\*.exe). Covers the two components that run as their own Windows services rather than under the supervisor (PostgreSQL as mdss_postgresql, RabbitMQ and Erlang as mdss_rabbitmq) and the tools administrators run by hand (pg_dump.exe, opswat.mdss.reencryption.cli.exe).

3. C:\ProgramData\OPSWAT-ONTAP-Connector\OPSWAT-ONTAP-Connector.exe, when installed on this host.

Rule 1 also covers the SMB sidecar reading customer files through smb-mounts and the remediation temp copies. Every service is a .NET 10 application that compiles code at run time, so memory scanning and instrumentation add latency to each process, and the supervisor restarts any child that exits (it polls every 10 seconds), so a blocked process becomes a restart loop. Executables, signers and ports: A.3 and A.4.

3.3 Scripts

Under script control, PowerShell protection and .NET Framework instrumentation, allow every .ps1 file in <INSTALL_DIR>\scripts\, and the powershell.exe that runs it, when started by the mdss service (rule 3.2.1), by msiexec.exe during install, upgrade or uninstall, or by an administrator from an elevated prompt. They are Windows PowerShell 5.1 scripts signed by OPSWAT, Inc. and launched with -ExecutionPolicy Bypass … -File, so key the rule on publisher or path, not on the execution policy. The installer additionally runs inline PowerShell commands (powershell.exe -Command), VBScript custom actions inside msiexec.exe (the UAC relaunch, WMI queries of ports, processes and services) and cmd.exe /c net stop … sc delete …. A.5 lists what each script does.

3.4 Services, scheduled task, firewall rules and registry

Allow the installer and the processes in 3.2 to create, change and delete:

• Windows services mdss, mdss_postgresql, mdss_rabbitmq, legacy mdss_mongodb, and opswat-ontap-connector.

• The scheduled task MDSS_InstallSharePointComponents (SharePoint On-Premises module only).

• Windows Firewall inbound rules whose display name starts with MetaDefender Storage Security:.

• Registry keys under HKLM\SOFTWARE\OPSWAT, HKLM\SOFTWARE\Ericsson\Erlang\ErlSrv, HKLM\SYSTEM\CurrentControlSet\Services\mdss* and …\Services\opswat-ontap-connector, the machine environment key HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment, the machine Trusted Root and Personal certificate stores, and (ONTAP Connector only) HKLM\SOFTWARE\WOW6432Node\Data ONTAP\Clustered Data ONTAP Antivirus Connector\v1.0\data-lifs.

Details: A.2 and A.6.

3.5 Network

• Inbound: 80 and 443 (WEB_PORT, WEB_PORT_SSL): console, REST API, storage webhooks and ONTAP Connector traffic, all through nginx.

• Loopback: all MDSS components talk to each other on 127.0.0.1. Do not filter loopback traffic for the processes in 3.2 (ports in A.4).

• Outbound:

  • Scan engines: MetaDefender Core (HTTP or HTTPS, default port 8008); api.metadefender.com:443 when MetaDefender Cloud is used.

  • OPSWAT: activation.dl.opswat.com:443 for license activation and, only when ENABLE_TELEMETRY=yes, for telemetry (OTLP over HTTPS). The previous telemetry endpoint mdss-telemetry.metadefender.com:4317 is used only when configured explicitly.

  • Protected storages: SMB 445; NFS 2049 and 111; SFTP 22; FTP 21 plus passive ports; 443 to the cloud storage endpoints (Amazon S3, Azure, Google Cloud, Alibaba, Oracle, Box, Microsoft Graph, SharePoint).

  • Customer infrastructure: domain controllers or LDAP servers (389, 636, 3268, 3269), the SMTP server, webhook targets, and an external PostgreSQL or RabbitMQ when configured.

3.6 Behaviours to recognise

These are normal for MDSS. Excluding the processes in 3.2 from the behavioural, anti-ransomware and child-process modules handles all of them; the list helps the security team recognise the alerts.

1. Bulk file operations on network shares. Remediation workflows delete, move, copy, rename and overwrite many files per minute, which triggers mass-delete, mass-rename, canary-file and "encryption-like write" heuristics.

2. Reading, and temporarily copying, malicious files before they are sent to MetaDefender Core or moved to quarantine.

3. Symbolic links to UNC paths. The SMB sidecar maps shares with WNetAddConnection2 and creates directory symbolic links under smb-mounts with CreateSymbolicLink, on demand, removing them after 300 seconds idle. Agents that follow reparse points scan, and may quarantine on, the customer's storage.

4. A service spawning PowerShell and management tools. At every start the mdss service runs powershell.exe -ExecutionPolicy Bypass, which spawns more PowerShell, sc.exe, icacls.exe, nginx.exe, psql.exe, node.exe and msiexec.exe, registers services and changes their logon account.

5. Start-up and installer changes to the machine. Every start imports CA certificates into the Trusted Root store, rewrites about 300 machine environment variables and broadcasts WM_SETTINGCHANGE. The installer relaunches itself elevated through UAC, queries WMI, adds firewall rules and a scheduled task, and writes state files to C:\ProgramData\OPSWAT. A support package runs systeminfo, route print and ipconfig /all and creates large archives under <INSTALL_DIR>.

3.7 Application control (allow-listing)

1. Publisher rule for OPSWAT, Inc.: all MDSS executables, libraries and PowerShell scripts, and the ONTAP Connector.

2. Publisher rule for Microsoft Corporation: the private .NET runtime, Garnet, PowerShell, msiexec.exe, sc.exe, icacls.exe.

3. Path rule for everything under <INSTALL_DIR>\, including SQL and JavaScript files. Covers the third-party runtimes (PostgreSQL, Erlang, RabbitMQ, Python, Node.js, MongoDB) whatever their signer, and nginx.exe, which is not signed by OPSWAT in 4.6.x. The folder is writable only by administrators, which is what makes a path rule acceptable.

If policy requires publisher rules for every file, enumerate the signers with the command in A.7 and add hash rules for the unsigned files it reports. The mdss service performs its own integrity check: it refuses to start an MDSS binary that is not signed by OPSWAT, or a runtime executable (dotnet.exe, GarnetServer.exe, python.exe) that carries no signature. Do not let the agent modify, wrap or re-sign these files.

4. OPSWAT ONTAP Connector (NetApp ONTAP)

A separate Windows service, usually on its own host next to the NetApp Antivirus Connector (ontapavc). Installed to C:\ProgramData\OPSWAT-ONTAP-Connector\ by MDSS or manually; the MDSS installer updates and removes it together with MDSS.

• Service: opswat-ontap-connector, executable OPSWAT-ONTAP-Connector.exe, signed by OPSWAT.

• Files: config.ini (rewritten through config.ini.tmp whenever the set of protected storages changes), data.json, logs\YYYYMMDD.log and YYYYMMDD_HHMMSS.log (or the configured LogPath). On uninstall, config.ini is backed up to C:\ProgramData\OPSWAT\ontap-connector-config.ini.bak.

• Named pipes: \pipe\ntapvsrq (scan requests from the NetApp AV Connector) and \pipe\ntapvscp (registration and verdicts), RPC over ncacn_np to 127.0.0.1.

• Registry: writes and deletes data-lif::<host> values under HKLM\SOFTWARE\WOW6432Node\Data ONTAP\Clustered Data ONTAP Antivirus Connector\v1.0\data-lifs, creating the key if it is missing. Since 4.6.0 this runs automatically every 15 seconds (StorageSyncIntervalInSeconds), not only at installation.

• Network: HTTP or HTTPS to the MDSS web port (80 or 443) for /api/webhook/realtime, /api/file/*, /api/health and /api/account (polled every 15 seconds). SMB 445 to the ONTAP shares.

• Behaviour: reads every file a user opens or modifies on the protected shares through \\?\UNC\… paths and posts it to MDSS. These files may be malicious. Optionally writes the Zone.Identifier alternate data stream (ZoneIdentifierFiles).

Exclude the folder, the process and its UNC file reads from on-access scanning. An agent that scans the network files this process opens scans, and may quarantine on, the ONTAP volume before MDSS returns a verdict, and it doubles the latency of every file open for every ONTAP user.

5. What to keep enabled

• Everything outside the paths and processes above. MDSS does not need the agent disabled.

• On-access scanning of files that administrators copy onto the server themselves, and network protection for destinations not listed in 3.5.

• Audit-only logging of the behaviours in 3.6, if the security team wants visibility. Only blocking and quarantine actions need to be excluded.

6. Verification

1. PowerShell start-up time, as administrator. Expect well under one second for the first command and one to three seconds for the second.

   Measure-Command { powershell.exe -NoProfile -NonInteractive -Command exit }

   cd "<INSTALL_DIR>\scripts"

   Measure-Command { powershell.exe -NoProfile -File .\mdss.ps1 -c load-ca }

2. Restart MDSS, read the step timings and confirm every process and service is up. Every step except database migrations should finish within seconds; a healthy restart takes two to four minutes.

   .\mdss.ps1 -c restart

   Get-Content "<INSTALL_DIR>\data\webclient\startup-log.txt"

   .\mdss.ps1 -c status

   Get-Service mdss, mdss_postgresql, mdss_rabbitmq

3. Check the agent console for detections referencing the paths in 3.1, the processes in 3.2 or the behaviours in 3.6. Any remaining detection means an exclusion is missing, or is by file name where a full path is required.

4. Scan a storage that holds a known-safe test file and, if remediation is used, run a workflow that moves a file to quarantine. Confirm in the MDSS audit log that both complete, and on the storage that nothing other than MDSS removed a file.

Appendix A. Detailed inventory

Use this appendix when your product has no wildcard or parent-process trust feature, or when policy requires each exclusion to be enumerated.

A.1 Installation folder layout

Outside the installation folder: C:\ProgramData\OPSWAT\ (mdss_upgrade_customer.env and mdss_upgrade_service_account.txt during upgrades, ontap-connector-config.ini.bak, mdss\smb-mounts\), C:\ProgramData\OPSWAT-ONTAP-Connector\, the service account's temp folder (3.1), and %TEMP%\MetaDefenderStorageSecurity-Setup-<timestamp>.log of the administrator who ran the installer.

A.2 Windows services, scheduled task and firewall rules

A.3 Executables

services.exe starts opswat.mdss.windows.service.exe, which starts powershell.exe for the start-up orchestration; that starts more powershell.exe instances, nginx.exe -s reload, psql.exe, node.exe and msiexec.exe. After orchestration the service starts and keeps alive every process below, restarting any that exits.

A.4 Microservices hosted by dotnet.exe, and loopback ports

If your product filters on the command line rather than the process path, allow dotnet.exe when its argument is one of these files (paths under <INSTALL_DIR>\services\; each service runs with its own folder as the working directory). Ports are bound to 127.0.0.1 unless the customer changed the listening IP.

Other loopback ports: 6002 and 7002 SMB sidecar (legacy Python sidecar: 6002 to 6006 and 7002 to 7006); 5432 PostgreSQL; 5672, 15672, 25672 and 4369 RabbitMQ, its management interface, Erlang distribution and port mapper; 6379 Garnet cache; 27017 and 27778 legacy MongoDB and its migration tool.

All storage providers (Amazon S3, Azure, Google Cloud, Alibaba, Oracle, Box, Microsoft Graph, SharePoint, SFTP, FTP, SMB and NFS) are libraries loaded in-process by the Discovery, Storages, Remediations and Scanning services; there are no per-provider processes. The Scanning service verifies the Authenticode signature of its providers with Windows before loading them. An agent that blocks or delays library loading, or that strips or alters signatures, prevents these services from starting.

A.5 PowerShell scripts

All scripts live in <INSTALL_DIR>\scripts\, are signed by OPSWAT, Inc., and run on Windows PowerShell 5.1 (.NET Framework). Agents that hook the .NET Framework through the APPDOMAIN_MANAGER_ASM and APPDOMAIN_MANAGER_TYPE machine environment variables inspect every one of these processes and can add minutes to each.

A.6 Registry keys

• HKLM\SOFTWARE\OPSWAT\MetaDefender Storage Security: InstallDir and installation state, written by the installer and read back on upgrade.

• HKLM\SYSTEM\CurrentControlSet\Services\mdss, mdss_postgresql, mdss_rabbitmq, mdss_mongodb, opswat-ontap-connector: service definitions, recovery options and logon account, written by the installer and the scripts.

• HKLM\SOFTWARE\Ericsson\Erlang\ErlSrv\1.1\MDSS_RabbitMQ: written by the RabbitMQ service registration.

• HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment: about 300 MDSS machine environment variables (ports, connection strings, feature flags, DOTNET_ROOT, ERLANG_HOME, RABBITMQ_*), rewritten at every service start.

• HKLM\SOFTWARE\Microsoft\SystemCertificates\ROOT\Certificates: customer CA certificates imported by mdss.ps1 -c load-ca. enable_https also uses the machine Personal (My) store temporarily.

• HKLM\SOFTWARE\WOW6432Node\Data ONTAP\Clustered Data ONTAP Antivirus Connector\v1.0\data-lifs: one data-lif::<host> value per protected NetApp SVM, maintained automatically by the ONTAP Connector.

A.7 Code signers

• OPSWAT, Inc.: opswat.mdss.windows.service.exe, opswat.mdss.smb.service.exe, opswat.mdss.reencryption.cli.exe, OPSWAT-ONTAP-Connector.exe, every opswat.*.dll, every .ps1 in <INSTALL_DIR>\scripts\, and the installer itself.

• Microsoft Corporation: dotnet.exe and the .NET runtime libraries under services\dotnet, GarnetServer.exe, powershell.exe, msiexec.exe, sc.exe, icacls.exe.

• Third-party runtime vendors (PostgreSQL build vendor, Erlang/OTP, RabbitMQ, Python Software Foundation, Node.js, MongoDB Inc.): the corresponding runtimes under services\.

• Not signed by OPSWAT: services\opswat.mdcs.webclient\nginx.exe, and possibly some third-party runtime files. Use path or hash rules for these.

Enumerate the real signers on the installed system before writing publisher rules, because third-party runtime versions change between MDSS releases:

Application control must also allow the PowerShell scripts in <INSTALL_DIR>\scripts\ and the SQL and JavaScript files in <INSTALL_DIR>\Utils\ and <INSTALL_DIR>\services\opswat.mdcs.mongomigrations\.

Appendix B. Changes since the 4.5.x article

If you applied the 4.5.x version of this list, update these items:

• SFTP sidecar removed. opswat.mdss.sftp.service, its loopback port 5003 and its firewall rule no longer exist. SFTP, FTP and all cloud providers run in-process inside the Discovery, Storages, Remediations and Scanning services.

• Telemetry endpoint. Telemetry goes to activation.dl.opswat.com:443 by default; mdss-telemetry.metadefender.com:4317 is used only when configured explicitly.

• Installer behaviour. The installer relaunches itself elevated through UAC, checks ports through WMI, runs inline PowerShell commands, and keeps upgrade state in C:\ProgramData\OPSWAT instead of %APPDATA%\OPSWAT\MetaDefender Storage Security\backup\. It also updates and uninstalls the ONTAP Connector.

• ONTAP Connector synchronisation. The connector polls /api/account every 15 seconds and rewrites config.ini and the data-lif registry values on its own. Its log folder is configurable (LogPath).

• Signed scripts. The PowerShell scripts carry an OPSWAT, Inc. Authenticode signature, so publisher-based script rules are possible. Blocked revocation checks for that signature are a separate cause of slow start-up (section 1).

• Environment variables. The machine environment holds about 300 MDSS variables, up from about 200.