Event-based real-time cloud functions examples
APIENDPOINT, APIKEY and STORAGECLIENTID need to be added as environment variables and should be accessible from the function. Please check Event-based handling for more details about these variables.
Amazon S3 lambda function setup
For creating a AWS lambda function, follow the official documentation: https://aws.amazon.com/getting-started/hands-on/run-serverless-code/
Function trigger needs to be set to all objects created events in order to receive notification only for newly added files.
Python function example:
import requestsimport osimport jsonfrom urllib.parse import unquotedef lambda_handler(event, context): for eventRecord in event['Records']: eventRecord['s3']['object']['key'] = unquote(eventRecord['s3']['object']['key'].replace("+", " ")) requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(eventRecord), 'storageClientId': os.getenv('STORAGECLIENTID', "") })Azure Blob function app setup
To deploy the Azure function app, please use the following Terraform script: https://github.com/OPSWAT/metadefender-k8s/tree/main/terraform/azure-function-docker
STORAGECLIENTID, APIKEY and APIENDPOINT variables should be configured on .tvars file:
resource_group_name = "" #The name of the resource group in which the function app will be created."service_plan_name = "" #The name of the app service planstorage_account_name = "" #The name of the storage account to be createddocker_registry_server_url = ""docker_registry_server_username = "" #optionaldocker_registry_server_password = "" #optionaldocker_image_name = ""docker_image_tag = ""AzureWebJobsBlobTrigger = "" #The storage account connection string that triggers the functionCONTAINERNAME = "" #The blob container that needs to be scannedfn_name_prefix = "" #function namelocation = "" #azure regionSTORAGECLIENTID = ""APIKEY = ""APIENDPOINT = ""Azure Blob Event Grid RTP configuration
For a detailed example, please use the example here: https://github.com/OPSWAT/metadefender-k8s/tree/main/terraform/CloudFunctions/Azure/webhook-notification
Event Notifications for Page and Append blob is not supported.
For Page and Append blobs, an event is sent as soon as the first block is committed to the storage, which can result in events being sent before the upload is complete.
Google Cloud function setup
The google.cloud.storage.object.v1.finalizedtrigger needs to be setup for the Cloud Function(v2), in order to process newly added objects.
Python function example:
mport functions_frameworkimport jsonimport requestsimport os# Triggered by a change in a storage bucket.cloud_eventdef hello_gcs(cloud_event): requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(cloud_event.data), 'storageClientId': os.getenv('STORAGECLIENTID', "") })Alibaba Cloud function setup
Follow the official Alibaba documentation for creating a compute function with OSS trigger: https://www.alibabacloud.com/help/en/function-compute/latest/configure-an-oss-trigger
When the function compute is created, it is necessary to specify the bucket to monitor and to subscribe to the following event oss:ObjectCreated:*
Python function example:
import oss2, json, osimport requestsdef handler(event, context): for eventRecord in json.loads(event)['events']: requests.post(os.getenv('APIENDPOINT', ""), headers={'ApiKey':os.getenv('APIKEY', "")}, json = {'metadata': json.dumps(eventRecord), 'storageClientId': os.getenv('STORAGECLIENTID', "") })Wasabi function setup
Follow the official Wasabi documentation to create an event notification: Event Notification (wasabi.com)
After setting up the event notification, it must establish a connection with a service capable of sending a request to MDSS.
The example described in the Wasabi documentation connects with AWS SNS: How do I configure Event Notifications on my Wasabi bucket using AWS SNS? which then can be used with AWS Lambda (see example above with Amazon S3 lambda function setup)
S3 Compatible function setup
There is a different way to create event based RTP for each S3 Compatible service.
In general most S3 Compatible services have event based notification which is similar to the Wasabi function setup.
Then a request needs to reach MDSS endpoint: http(s)://{baseurl}/api/webhook/realtime with the body:
{ "storageClientId": \"{Storage Client Id}\", "metadata": "{\"s3\": { \"object\": {\"key\": \"{Object Path}\" }}}"}
