Event-Based Real-Time Cloud Functions Examples

Amazon S3 Lambda Function Setup

This setup uses AWS Lambda to react to S3 object creation events.

Setup steps:

1. Create an AWS Lambda Function:

   1. Follow the official AWS documentation to create a new Lambda function: https://aws.amazon.com/getting-started/hands-on/run-serverless-code/ and choose Python as the runtime.

2. Configure the S3 Trigger:

   1. In the lambda function configuration, add an S3 trigger.
   2. Select the S3 bucket you want to monitor.
   3. Set the "Event type" or "All object create events" (or a similar option like s3:ObjectCreated:*). This ensures the function only triggers for newly added or overwritten files

3. Deploy the Function Code - use the Python code example below as your Lambda function handler.

4. Configure Environment Variables - in the lambda function's configuration settings (Under "Environment variables") add APIENDPOINT, APIKEY, and STORAGECLIENTID with their respective values.

import requests

import os

import json

from urllib.parse import unquote

​

def lambda_handler(event, context):

    for eventRecord in event['Records']:

        eventRecord['s3']['object']['key'] = unquote(eventRecord['s3']['object']['key'].replace("+", " "))

        requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(eventRecord), 'storageClientId': os.getenv('STORAGECLIENTID', "") })

Azure Blob Function app setup

This setup uses Azure Functions triggered by new blobs in Azure Blob Storage. The example focuses on deployment via Terraform.

Setup using Terraform:

1. Get the Terraform Script: Clone or download the script from the OPSWAT repository: https://github.com/OPSWAT/metadefender-k8s/tree/main/terraform/azure-function-docker
2. Configure variables in the .tvars file: Create a terraform.tfvars file (or similar) and define the required variables for MDSS: APIENDPOINT, APIKEY, and STORAGECLIENTID.
3. Deploy: Run terraform init, terraform plan, and terraform apply from the directory containing the script and your .tfvars file - this will provision the Azure Function App, configure the trigger, and set the MDSS environment variables.

resource_group_name = "" #The name of the resource group in which the function app will be created."

service_plan_name = "" #The name of the app service plan

storage_account_name = "" #The name of the storage account to be created

docker_registry_server_url = ""

docker_registry_server_username = "" #optional

docker_registry_server_password = "" #optional

docker_image_name = ""

docker_image_tag = ""

AzureWebJobsBlobTrigger = "" #The storage account connection string that triggers the function

CONTAINERNAME = "" #The blob container that needs to be scanned

fn_name_prefix = "" #function name

location = "" #azure region

STORAGECLIENTID  = ""

APIKEY = ""

APIENDPOINT = ""

Azure Blob Event Grid RTP Configuration (Alternative/Advanced)

• for potentially higher scale or different event filtering, you can use Azure Event Grid to trigger your function instead of the direct Blob trigger.
• Refer to this example for detailed configuration: https://github.com/OPSWAT/metadefender-k8s/tree/main/terraform/CloudFunctions/Azure/webhook-notification

Google Cloud Storage & Google Cloud Functions

This setup uses Google Cloud Functions (Gen 2 recommended) triggered by Cloud Storage events.

Setup Steps:

1. Create a Cloud Function (Gen 2): Use the Google Cloud Console or gcloud CLI to create a new Cloud Function (Generation 2).

2. Configure the Trigger

   • Set the "Trigger type" to "Cloud Storage".
   • Set the "Event type" to google.cloud.storage.object.v1.finalizedtrigger . This event fires only after an object has been successfully created or replaced in the bucket.
   • Select the Cloud Storage bucket to monitor.

3. Deploy the Function Code - use the Python code example below. Ensure functions-framework is listed in your requirements.txt.

4. Configure Environment Variables: During deployment (via Console UI or gcloud deploy command), set the APIENDPOINT, APIKEY, and STORAGECLIENTID environment variables.

import functions_framework

import json

import requests

import os

​

# Triggered by a change in a storage bucket

@functions_framework.cloud_event

def hello_gcs(cloud_event):

   requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(cloud_event.data), 'storageClientId': os.getenv('STORAGECLIENTID', "") })

Alibaba Cloud Object Storage Service (OSS) & Function Compute

This setup uses Alibaba Cloud Function Compute triggered by OSS events.

1. Create a Function Compute Function: Follow the official Alibaba Cloud documentation to create a compute function: https://www.alibabacloud.com/help/en/function-compute/latest/configure-an-oss-trigger . Choose a Python runtime.

2. Configure the OSS Trigger:

   1. Specify the OSS bucket you wish to monitor.
   2. Subscribe to the oss:ObjectCreated:* event type to trigger the function for all object creation events
   3. Specify the bucket to monitor and subscribe toPython Function Example:

3. Deploy the Function Code - use the Python example below as your function handler.

4. Configure Environment Variables: In the Function Compute service configuration, define the APIENDPOINT, APIKEY, and STORAGECLIENTID environment variables for your function.

import oss2, json, os

import requests

​

def handler(event, context):

    for eventRecord in json.loads(event)['events']:

        requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(eventRecord), 'storageClientId': os.getenv('STORAGECLIENTID', "") })

Wasabi Cloud Storage Function Setup

Wasabi uses an S3-compatible API but often requires integrating with another service (like AWS SNS/Lambda) for compute capabilities.

Setup Steps:

1. Configure Event Notifications in Wasabi: Follow the official Wasabi documentation to set up bucket event notifications: https://docs.wasabi.com/v1/docs/event-notifications-bucket

2. Choose a Target Service: Configure Wasabi notifications to send events to a service capable of making HTTP POST requests. A common pattern is to send notifications to:

   • AWS Simple Notification Service (SNS): See Wasabi's guide: How do I configure Event Notifications on my Wasabi bucket using AWS SNS?

3. Process Notifications:

   • If using AWS SNS, configure it to trigger an AWS Lambda function (similar to the AWS S3 setup described in Section 1).
   • The Lambda function will receive the event notification (originally from Wasabi, forwarded by SNS), extract the relevant object information, and send the request to the MDSS APIENDPOINT.

4. Configure Lambda - make sure the Lambda function has the necessary code (adapt the AWS S3 example, checking the event structure from SNS) and the APIENDPOINT, APIKEY, STORAGECLIENTID environment variables set. The event payload structure might differ slightly when coming via SNS compared to a direct S3 trigger.

Other S3-Compatible function setup

Many other storage services offer S3-compatible APIs and event notification mechanisms (e.g., MinIO, Ceph RGW).

General Configuration Approach

1. Enable Event Notifications - consult your specific S3-compatible service's documentation to enable event notifications for object creation (similar to s3:ObjectCreated:*).

2. Set Up a Target/Webhook - configure the notifications to be sent to a target service. This might be:

   • A webhook endpoint (if the service supports direct HTTP/S POST for notifications).
   • A message queue (like Kafka, RabbitMQ, NATS).
   • A specific function-as-a-service platform integrated with the storage.

3. Implement the Processing Logic - create a function, service, or script that:

   • Receives the event notification.
   • Parses the event data to extract the object path/key
   • Constructs the required JSON payload (see below)
   • Sends an HTTP POST request to your MDSS APIENDPOINT

4. Configure Environment Variables - make sure APIENDPOINT, APIKEY, and STORAGECLIENTID are accessible to your processing logic.

Required MDSS Webhook Request

Your processing logic must send an HTTP POST request to: http(s)://{your_mdss_host}/api/webhook/realtime

With headers:

• ApiKey: {your_mdss_api_key}
• Content-Type: application/json

And a JSON body structured like this:

{

    "storageClientId": [Storage Client Id],

    "metadata": "{'s3': { 'object': {'key': [Object Path] }}}"

}

• Replace YOUR_MDSS_STORAGE_CLIENT_ID with the value from your environment variable
• Replace YOUR_OBJECT_PATH/KEY with the actual key/path of the created object, extracted from the event notification
• The value of the "metadata" field must be a JSON string , not a nested JSON object. The example shows the simplest required structure. You might include more S3-like metadata within the stringified JSON if needed, matching the structure MDSS expects (often based on the AWS S3 event structure). Check MDSS documentation for precise metadata requirements if defaults are insufficient.