Efficient Email Automation: Harnessing S3 Events with SQS, Lambda, and SNS

In today’s digital landscape, timely communication is paramount for businesses to stay competitive. Whether it’s notifying users about new content updates, processing customer inquiries, or triggering internal workflows, automating email notifications has become a cornerstone of modern applications.

Amazon Simple Storage Service (S3) is a powerful tool for storing and managing data in the cloud. However, leveraging S3 to trigger email notifications in response to file uploads or modifications can be challenging. This is where infrastructure-as-code tools like Terraform shine.

In this article, we’ll explore how to streamline email automation using S3 event notifications combined with Terraform to provision all the necessary AWS resources, including Simple Queue Service (SQS), Lambda, and Simple Notification Service (SNS). We’ll delve into the architecture, implementation steps, and best practices for building a robust and scalable email notification system on AWS entirely using Terraform.

By the end of this guide, you’ll have a clear understanding of how to harness the power of Terraform and AWS services to automate email notifications effectively, enabling you to enhance user engagement, streamline business processes, and drive operational efficiency.

Let’s structure the process of triggering email notifications upon uploading data to an S3 bucket in a clearer and organized manner:

1. Infrastructure Setup:
   Use Terraform to provision the necessary AWS resources:
   - S3 Bucket: Create an S3 bucket to store the data.
   - SQS Queue: Set up an SQS queue to handle notifications.
   - Lambda Function: Create a Lambda function to process SQS messages and trigger email notifications.
   - SNS Topic: Provision an SNS topic to publish email notifications.
   - IAM Roles and Policies: Define roles and policies to grant necessary permissions for Lambda function execution, S3, SQS, and SNS operations.
2. Configure S3 Bucket Notifications:
   Set up S3 bucket notifications to trigger events whenever objects are created or modified in the bucket.
   Configure the notifications to send messages to the SQS queue for further processing.
3. Lambda Function Logic:
   - Develop the Lambda function code to:
   - Receive messages from the SQS queue.
   - Process the received messages, such as extracting relevant information.
   - Trigger email notifications using the SNS topic.
4. Subscribe to SNS Topic: Subscribe an email address (or multiple email addresses) to the SNS topic to receive notifications.
   Ensure that the email subscription confirmation process is completed.
5. Testing:
   Upload data to the S3 bucket to trigger the configured S3 bucket notifications.
   Verify that the Lambda function processes the SQS messages and triggers email notifications via SNS.
   Check the subscribed email inbox for the notifications.
6. Monitoring and Maintenance:
   Monitor the system to ensure proper functioning of the notification mechanism.
   Regularly review and update configurations as needed, such as updating email subscriptions or modifying notification triggers.

Provider Configuration:

provider "aws" {
  region = var.region
}

• This section configures the AWS provider for Terraform, specifying the AWS region where the resources will be created. The region is provided dynamically using the var.region variable.

Terraform Backend Configuration:

terraform {
  backend "s3" {
    bucket = "frank-terraform-tfstate"
    key    = "terraform/"
    region = "us-east-1"
  }
}

• Here, Terraform’s backend is configured to store the state file in an S3 bucket named “frank-terraform-tfstate” in the “us-east-1” region. This allows for centralized state management and collaboration among team members.

IAM Policy for S3 and SQS:

resource "aws_iam_policy" "s3_sqs" {
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqss3id",
  "Statement": [
    {
      "Sid": "sqss3",
      "Effect": "Allow",
      "Action": [
        "SQS:*"
      ],
      "Resource": "${aws_sqs_queue.event_queue.arn}",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "${aws_s3_bucket.bucket.arn}" 
        },
        "StringEquals": {
          "aws:SourceAccount": "400143264072"
        }
      }
    }
  ]
}
POLICY
}

• This defines an IAM policy named “s3_sqs” that allows actions on SQS queues based on certain conditions related to S3 bucket ARN and AWS account ID.

IAM Role for Lambda Execution:

resource "aws_iam_role" "lambda_exec" {
  name = var.iam_role_name
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect = "Allow",
      Principal = {
        Service = "lambda.amazonaws.com"
      },
      Action = "sts:AssumeRole"
    }]
  })
  # Attach policies
  inline_policy {
    name = "lambda_sqs_policy"
    policy = jsonencode({
      Version = "2012-10-17",
      Statement = [{
        Effect   = "Allow",
        Action   = "sqs:*",
        Resource = aws_sqs_queue.event_queue.arn
      }]
    })
  }
  inline_policy {
    name = "AWSLambdaBasicExecutionRole"
    policy = jsonencode(
      {
        "Version" : "2012-10-17",
        "Statement" : [
          {
            "Effect" : "Allow",
            "Action" : [
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:PutLogEvents"
            ],
            "Resource" : "*"
          }
        ]
      }
    )
  }
  inline_policy {
    name = "AmazonSNSAccess"
    policy = jsonencode({
      "Version" : "2012-10-17",
      "Statement" : [
        {
          "Action" : [
            "sns:*"
          ],
          "Effect" : "Allow",
          "Resource" : "*"
        }
      ]
      }
    )
  }
  inline_policy {
    name = "lambda_s3_policy"
    policy = jsonencode({
      Version = "2012-10-17",
      Statement = [{
        Effect = "Allow",
        Action = [
          "s3:*"
        ],
        Resource = [
          "${aws_s3_bucket.bucket.arn}",
          "${aws_s3_bucket.bucket.arn}/*"
        ]
      }]
    })
  }
}

• An IAM role named “lambda_exec” is created to provide permissions for the Lambda function to interact with other AWS services.

IAM Role Policy Attachment:

data "aws_iam_policy_document" "queue" {
  statement {
    effect = "Allow"

    principals {
      type        = "*"
      identifiers = ["*"]
    }

    actions   = ["sqs:SendMessage"]
    resources = ["${aws_sqs_queue.event_queue.arn}"]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = ["${aws_s3_bucket.bucket.arn}"]
    }
  }
}

• This attaches the IAM policy (defined earlier) to the IAM role created for Lambda execution.

resource "aws_iam_role_policy_attachment" "attach" {
  role       = aws_iam_role.lambda_exec.name
  policy_arn = aws_iam_policy.s3_sqs.arn
}

• An S3 bucket is provisioned with the specified name. This bucket will be used to store files and trigger events for the email notification system.

S3 Bucket Creation:

resource "aws_s3_bucket" "bucket" {
  bucket = var.bucket_name
}

• A S3 bucket is provisioned to store files and trigger events for the email notification system.

SQS Queue Creation:

resource "aws_sqs_queue" "event_queue" {
  name = var.sqs_queue_name
}

• An SQS queue is created to receive events triggered by actions on the S3 bucket.

S3 Bucket Notification Configuration:

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = aws_s3_bucket.bucket.id

  queue {
    queue_arn = aws_sqs_queue.event_queue.arn
    events    = ["s3:ObjectCreated:*"]
    # filter_suffix = ".log"
  }
}

• Configuration for S3 bucket notifications is defined to trigger events for file uploads to the bucket.

SQS Queue Policy:

resource "aws_sqs_queue_policy" "queue_policy" {
  queue_url = aws_sqs_queue.event_queue.id
  policy    = data.aws_iam_policy_document.queue.json
}

• A policy is attached to the SQS queue to allow S3 to send messages to the queue.

Lambda Function Creation:

resource "aws_lambda_function" "sqs_poller" {
  function_name    = var.lambda_function_name
  runtime          = var.lambda_function_runtime
  handler          = "lambda_function.lambda_handler"
  filename         = var.lambda_function_filename
  role             = aws_iam_role.lambda_exec.arn
  timeout          = 30
  environment {
    variables = {
      SNS_TOPIC_ARN = aws_sns_topic.notification_topic.arn
    }
  }

}

• This defines the Lambda function responsible for processing SQS messages and triggering email notifications.

The Lambda will be executing the python code on python 3.8 interpretor.

This Python code is intended to be used as a Lambda function handler. It interacts with AWS Simple Notification Service (SNS) to publish a message to a specified SNS topic. Let’s break down the code:

import os
import boto3

• The os module is imported to access environment variables.
• The boto3 module is imported to interact with AWS services using Python.

# Initialize SNS client
sns = boto3.client('sns')

• This line initializes an SNS client using the boto3.client() method. The client is used to interact with SNS topics and publish messages to them.

def lambda_handler(event, context):

• This function is the entry point for the Lambda execution. It is the handler function that AWS Lambda calls when the Lambda function is invoked.

    # Get SNS topic ARN from environment variable
    sns_topic_arn = os.environ['SNS_TOPIC_ARN']

• This line retrieves the ARN of the SNS topic from the Lambda function’s environment variables. It accesses the SNS_TOPIC_ARN environment variable using os.environ dictionary.

    # Publish email message to SNS topic
    response = sns.publish(
        TopicArn=sns_topic_arn,
        Message='File Uploaded to S3 Successfully. Lambda function executed successfully.'
    )

• This code block publishes a message to the specified SNS topic. It uses the sns.publish() method to send a message to the SNS topic with the ARN retrieved earlier. The message content is specified as a string.

    print("Email notification sent via SNS.")

• This line prints a message to indicate that the email notification has been sent via SNS.

    return {
        'statusCode': 200,
        'body': 'Email notification sent via SNS.'
    }

• Finally, this block returns a dictionary containing the response status code (200 for success) and a message indicating that the email notification has been sent via SNS. This response will be returned to the invoker of the Lambda function.

Lambda Function Event Source Mapping:

resource "aws_lambda_event_source_mapping" "event_source_mapping" {
  event_source_arn = aws_sqs_queue.event_queue.arn
  function_name    = aws_lambda_function.sqs_poller.arn
  enabled          = true
}

• An event source mapping is created to link the Lambda function with the SQS queue, enabling it to process incoming messages.

SNS Topic Creation:

resource "aws_sns_topic" "notification_topic" {
  name = var.sns_topic_name
}

• An SNS topic is provisioned to publish email notifications.

SNS Topic Subscription:

resource "aws_sns_topic_subscription" "email_subscription" {
  topic_arn = aws_sns_topic.notification_topic.arn
  protocol  = var.sns_protocol
  endpoint  = var.sns_topic_email
}

• An email subscription is added to the SNS topic to receive notifications.

You need to subscribe the sns subscription in the respective email which you have mentioned.

Lambda Event Source Mapping:

resource "aws_lambda_event_source_mapping" "event_source_mapping" {
  event_source_arn = aws_sqs_queue.event_queue.arn
  function_name    = aws_lambda_function.sqs_poller.arn
  enabled          = true
}

Terraform resource establishes an event source mapping between an SQS queue and a Lambda function

Lambda Function Event Invoke Configuration:

resource "aws_lambda_function_event_invoke_config" "event_invoke_config" {
  function_name = aws_lambda_function.sqs_poller.function_name

  destination_config {
    on_failure {
      destination = aws_sns_topic.notification_topic.arn
    }

    on_success {
      destination = aws_sns_topic.notification_topic.arn
    }
  }
}

• Configuration is set for the Lambda function to invoke SNS on both success and failure events.

This Terraform configuration sets up the infrastructure required for the email notification system, including storage, message queuing, message processing, and notification publishing. Each resource plays a specific role in the overall architecture, interconnected to facilitate the flow of events and actions.

Upload the data to s3 bucket. Here i have uploaded a test.png object.

After uploading the data to s3 you can now check the event generated in the email.