Event-based real-time cloud functions examples

Amazon S3 lambda function setup

For creating a AWS lambda function, follow the official documentation: https://aws.amazon.com/getting-started/hands-on/run-serverless-code/

Function trigger needs to be set to all objects created events in order to receive notification only for newly added files.

Python function example:

import requests

import os

import json

from urllib.parse import unquote

​

def lambda_handler(event, context):

    for eventRecord in event['Records']:

        eventRecord['s3']['object']['key'] = unquote(eventRecord['s3']['object']['key'].replace("+", " "))

        requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(eventRecord), 'storageClientId': os.getenv('STORAGECLIENTID', "") })

Azure Blob function app setup

To deploy the Azure function app, please use the following Terraform script: https://github.com/OPSWAT/metadefender-k8s/tree/main/terraform/azure-function-docker

STORAGECLIENTID, APIKEY and APIENDPOINT variables should be configured on .tvars file:

resource_group_name = "" #The name of the resource group in which the function app will be created."

service_plan_name = "" #The name of the app service plan

storage_account_name = "" #The name of the storage account to be created

docker_registry_server_url = ""

docker_registry_server_username = "" #optional

docker_registry_server_password = "" #optional

docker_image_name = ""

docker_image_tag = ""

AzureWebJobsBlobTrigger = "" #The storage account connection string that triggers the function

CONTAINERNAME = "" #The blob container that needs to be scanned

fn_name_prefix = "" #function name

location = "" #azure region

STORAGECLIENTID  = ""

APIKEY = ""

APIENDPOINT = ""

Azure Blob Event Grid RTP configuration

For a detailed example, please use the example here: https://github.com/OPSWAT/metadefender-k8s/tree/main/terraform/CloudFunctions/Azure/webhook-notification

Google Cloud function setup

The google.cloud.storage.object.v1.finalizedtrigger needs to be setup for the Cloud Function(v2), in order to process newly added objects.

Python function example:

mport functions_framework

import json

import requests

import os

​

# Triggered by a change in a storage bucket

@functions_framework.cloud_event

def hello_gcs(cloud_event):

   requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(cloud_event.data), 'storageClientId': os.getenv('STORAGECLIENTID', "") })

Alibaba Cloud function setup

Follow the official Alibaba documentation for creating a compute function with OSS trigger: https://www.alibabacloud.com/help/en/function-compute/latest/configure-an-oss-trigger

When the function compute is created, it is necessary to specify the bucket to monitor and to subscribe to the following event oss:ObjectCreated:*

Python function example:

import oss2, json, os

import requests

​

def handler(event, context):

    for eventRecord in json.loads(event)['events']:

        requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(eventRecord), 'storageClientId': os.getenv('STORAGECLIENTID', "") })

Wasabi function setup

Follow the official Wasabi documentation to create an event notification: Event Notification (wasabi.com)

After setting up the event notification, it must establish a connection with a service capable of sending a request to MDSS.

The example described in the Wasabi documentation connects with AWS SNS: How do I configure Event Notifications on my Wasabi bucket using AWS SNS? which then can be used with AWS Lambda (see example above with Amazon S3 lambda function setup)

S3 Compatible function setup

There is a different way to create event based RTP for each S3 Compatible service.

In general most S3 Compatible services have event based notification which is similar to the Wasabi function setup.

Then a request needs to reach MDSS endpoint: http(s)://{baseurl}/api/webhook/realtime with the body:

{

    "storageClientId": \"{Storage Client Id}\",

    "metadata": "{\"s3\": { \"object\": {\"key\": \"{Object Path}\" }}}"

}