Self-hosted shared services

Usage

1. Basic deployment with self-hosted shared services
2. Advanced deployment with self-hosted shared services

Overview

This guide provides a detailed procedure for deploying shared services (database, message broker, and cache) in a self-hosted environment, enabling you to implement one of the deployment options previously mentioned.

Basic deployment with self-hosted shared services

In this setup, we aim to segregate shared services from a basic single-instance deployment to allocate dedicated resources to them. Linux is the preferred operating system for this approach, as some services may not be compatible with Windows without some type of virtualization. These external services can be installed together on a single virtual machine (VM).

Advanced deployment with self-hosted shared services

In an advanced deployment designed to manage larger workloads and enhance reliability, we install shared services (database, message broker, and cache) on separate instances. This setup ensures that each service maintains high availability and scalability.

The key distinction between basic and advanced deployments in the context of self-hosted shared services lies in the incorporation of high availability for one or more of these services, as well as the extent to which scalability is implemented. For instance, certain workloads might require high availability for the database but do not necessitate a cluster of databases with partitioning (sharding).

At a glance, the process involves the following high-level actions:

1. Prepare system by updating system packages
2. Database deployment: MongoDB
3. Message Broker Deployment: RabbitMQ
4. Cache Deployment: Redis
5. Configure Storage Security to connect to the shared services
6. Start Storage Security using self-hosted shared services
7. (For advanced deployments) Scalability & High Availability

Update system packages

Before beginning the installation process, it's important to update your system's package repository to ensure you are installing the latest versions of the required packages along with any necessary dependencies. You can update your system's package repository with the following commands:

sudo apt update

sudo apt upgrade

This instruction is applicable to Ubuntu and Debian-based systems. For other Linux distributions, please use their respective package managers. For example, you would use yum on CentOS systems.

Database deployment: MongoDB

Before deploying MongoDB in a production environment, consider the Production Notes document which offers performance considerations and configuration recommendations for production MongoDB deployment

1. Add MongoDB Repository: MongoDB provides official repositories for various Linux distributions. Add the MongoDB repository to your system's package manager to simplify installation and updates.

For Ubuntu, you can add the MongoDB repository with the following commands:

wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list

For other distributions, refer to the official MongoDB documentation for instructions.

2. Install MongoDB: Once the repository is added, you can install MongoDB using the package manager. Run the following command:

sudo apt update

sudo apt install -y mongodb-org

This command will install the MongoDB packages and its dependencies.

3. Start MongoDB: After installation, start the MongoDB service using the following command:

sudo systemctl start mongod

You can also enable MongoDB to start on boot with:

sudo systemctl enable mongod

4. Verify Installation: To verify that MongoDB has been installed and is running correctly, you can check the service status:

sudo systemctl status mongod

If MongoDB is running, you should see output indicating that the service is active and running.

5. Access MongoDB: MongoDB by default listens on port 27017. Make sure that this port is accessible from the MDSS instance. You can access the MongoDB shell by typing:

mongo

This will open the MongoDB shell where you can start interacting with your MongoDB instance.

Message Broker Deployment: RabbitMQ

1. Install RabbitMQ: Install RabbitMQ using your package manager. On Ubuntu, you can use the following commands:

sudo apt install rabbitmq-server

For other distributions, refer to the RabbitMQ documentation for installation instructions specific to your platform.

2. Start and Enable RabbitMQ Service: Start the RabbitMQ service and enable it to start on boot:

sudo systemctl start rabbitmq-server

sudo systemctl enable rabbitmq-server

3. Configure RabbitMQ: RabbitMQ should now be running. By default, it listens on localhost. To allow access from other machines on the same network, you may need to modify the RabbitMQ configuration to listen on the VM's network IP address. Edit the RabbitMQ configuration file, usually located at /etc/rabbitmq/rabbitmq.conf, and add the following line:

listeners.tcp.1 = 0.0.0.0:5672

This will make RabbitMQ listen on all network interfaces on port 5672.

4. Firewall Configuration: Ensure that the firewall on the Linux VM allows inbound connections on the RabbitMQ port (default is 5672). You can use ufw on Ubuntu or firewalld on CentOS/RHEL to configure the firewall.
5. Access RabbitMQ Management Interface (Optional): RabbitMQ comes with a web-based management interface. If you want to access it from a web browser on your local network, you need to enable the RabbitMQ management plugin. Run the following command:

sudo rabbitmq-plugins enable rabbitmq_management

You can then access the management interface from a web browser by navigating to http://<your_vm_ip>:15672. Log in with the default credentials (guest/guest), or configure your own credentials in RabbitMQ.

Cache Deployment: Redis

1. Install Redis

sudo apt install redis-server

2. Configure Redis

Open the Redis configuration file for editing:

sudo nano /etc/redis/redis.conf

Adjust the configuration as needed. At the minimum, ensure:

• Redis is bound to the VM's IP address or to all interfaces (bind 0.0.0.0 or bind VM_IP).

• Remote connections are allowed (protected mode disabled):

  • protected-mode no

• Optionally, set a password for Redis authentication.

4. Test Redis Connectivity

Verify that Redis is listening on the correct interface and port

sudo netstat -tuln | grep redis

Test connectivity from the local VM:

redis-cli

5. Configure Firewall (if applicable):

   1. If there's a firewall running on the VM, ensure it allows incoming connections on the Redis port (default: 6379).

Configure Storage Security to connect to the external services

After deploying MongoDB, the connection string can be added in the MDSS configuration file /etc/mdss/customer.env like this:

MONGO_URL=mongodb://<username>:<insertYourPassword>@<address>:27017/MDCS

After deploying Redis, its endpoint can be added in the MDSS configuration file /etc/mdss/customer.env like this:

CACHE_SERVICE_URI=<redis_hostname_or_ip>:6379

CACHE_SERVICE_URL=<redis_hostname_or_ip>

CACHE_SERVICE_PORT=6379

CACHE_SERVICE_URL and CACHE_SERVICE_PORT also need to be configured for MDSS to check connectivity to the service before starting up.

After deploying RabbitMQ, its endpoint can be added in the MDSS configuration file /etc/mdss/customer.env like this:

RABBITMQ_URI=amqps://<rabbitmq_hostname_or_ip>:5671

RABBITMQ_HOST=<rabbitmq_hostname_or_ip>

RABBITMQ_PORT=5671

RABBITMQ_HOST and RABBITMQ_PORT also need to be configured for MDSS to check connectivity to the service before starting up.

Start MetaDefender Storage Security with Self-Hosted Shared Services

After MongoDB, RabbitMQ and Redis are all deployed, configured and accessible, MDSS can be started/restarted to apply the changes and use the new services:

# to start

sudo mdss -c start

# to restart

sudo mdss -c restart

Scalability and High Availability for Self-Hosted Shared Services

To achieve scalability and high availability, specialized guides are available for each service (database, message broker, and cache).

For MongoDB, detailed resources and best practices are available to guide you through setting up scalable and highly available environments.

• Database Scaling
• A Guide To Horizontal Vs Vertical Scaling

For RabbitMQ, you can refer to the official clustering guide available at:Clustering Guide | RabbitMQ.

For Redis, guidance on deploying in cluster mode to enhance scalability can be found at: Scale with Redis Cluster.

For further information or specific inquiries, please feel free to reach out to our support team.

Useful resources

More details can be found in their official documentation regarding deployment and configuration:

MongoDB: Install MongoDB Community Edition on Linux Redis: Install Redis RabbitMQ: Installing RabbitMQ | RabbitMQ