Process SQS messages with AWS Lambda

AWS Lambda is a cost-effective way to execute functions for high performance, low latency, and pay-as-you-go pricing model. AWS services that generate events can be used as an event source for Lambda functions. Combining Lambda with these services offers continuous scaling and built-in fault tolerance.

In this article, we will set up an AWS Lambda function to process messages from Amazon SQS.

Triggering Lambda from SQS

Lambda uses five parallel long-polling connections to request batches. When messages are available in the queue, Lambda invokes the function. The function will remove messages from the queue when batch is completed successfully. If there are more messages available, Lambda increases the number of processes that are reading batches. This automatic scaling behavior allows Lambda to scale up and process a higher number of messages.

One key point about this polling mechanism is, if there are no messages available in the queue, it can lead the NumberOfEmptyReceives metric to raise. If you have a low frequency of messages, it is worth monitoring the situation.

Creating Sample Application

Thanks to AWS SAM, there is an easy way to create a sample Lambda function that uses an SQS queue as an event source. With the help of SAM, it is also possible to create SQS queue from template.yml file.

$ sam init --runtime nodejs14.x --app-template quick-start-sqs --name sqs-lambda-app

By default, Amazon SQS creates a standard queue. You can specify additional parameters inside SimpleQueue resource in the template.yml file. In this example, we set queue name as lambda-test. Also, don’t forget to set FunctionName since we will use it later to check logs.

# This is the SAM template that represents the architecture of your serverless application
# https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-basics.html

# The AWSTemplateFormatVersion identifies the capabilities of the template
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/format-version-structure.html
AWSTemplateFormatVersion: 2010-09-09
Description: >-
  sqs-lambda-app

# Transform section specifies one or more macros that AWS CloudFormation uses to process your template
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/transform-section-structure.html
Transform:
- AWS::Serverless-2016-10-31

# Resources declares the AWS resources that you want to include in the stack
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resources-section-structure.html
Resources:
  # This is an SQS queue with all default configuration properties. To learn more about the available options, see
  # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html
  SimpleQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: "lambda-test"

  # This is the Lambda function definition associated with the source code: sqs-payload-logger.js. For all available properties, see
  # https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
  SQSPayloadLogger:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: SQSPayloadLogger
      Description: A Lambda function that logs the payload of messages sent to an associated SQS queue.
      Runtime: nodejs14.x
      Handler: src/handlers/sqs-payload-logger.sqsPayloadLoggerHandler
      # This property associates this Lambda function with the SQS queue defined above, so that whenever the queue
      # receives a message, the Lambda function is invoked
      Events:
        SQSQueueEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt SimpleQueue.Arn
      MemorySize: 128
      Timeout: 25 # Chosen to be less than the default SQS Visibility Timeout of 30 seconds
      Policies:
        - AWSLambdaBasicExecutionRole

Now, it is time to build and deploy our sample application:

$ sam build

# Use --guided option to deploy for the first time
$ sam deploy --guided

Let’s send a message to lambda-test queue that we just created with the AWS SAM template.

$ aws sqs send-message --queue-url "https://sqs.eu-west-1.amazonaws.com/xxxxxx/lambda-test" --message-body "hello from sqs-lambda trigger"

Let’s check our Lambda function log with sam logs command, Lambda function should process the message we just sent:

$ sam logs -n SQSPayloadLogger

2021/08/29/[$LATEST]0d137fd7aea444e782793cc43893ea31 2021-08-29T21:47:56.017000 START RequestId: 32163cb2-4d34-55b0-ad82-65144ff0daf0 Version: $LATEST
2021/08/29/[$LATEST]0d137fd7aea444e782793cc43893ea31 2021-08-29T21:47:56.088000 2021-08-29T21:47:56.068Z	32163cb2-4d34-55b0-ad82-65144ff0daf0	INFO	{"Records":[{"messageId":"7ab6789b-dd18-4b11-9121-077b8f938be0","receiptHandle":"AQEBgY5rDArrDbIrjWcG/n2K55v8auRkgSJGfH/enkuJjLytKrRsWeJXwDvqB3kRyK8Ay9RgVAPwxxgZKJ8vqzgxiCSs2au4GXR9gam40B3l4OVBLwUlbgmNbtMoFuOpnISzJMkC1JyQyo4Q/UKo6dUO2ihoeG/MDnG6oTjmZi/JSNfflru+KCcvlodz67U2fAh7FRGXje6nBGX8SeLvO2MUNwfEoQUpEn6oC4/VSbsQESF9lQi4VNTAWsnoV354La32rmn1m4TRDhDqCszEI/W22eFBRVMuH/WVoU9dQnCe+oclA3MysxL7tGhb/xkHR/ZlYjO7jXH+o9ieSqO/bWYyofUYJMvFZjNitVtqbagL0hHsxfyM38g44gTK1PaWBgo3","body":"hello from sqs-lambda trigger","attributes":{"ApproximateReceiveCount":"1","SentTimestamp":"1630273675920","SenderId":"sender_id","ApproximateFirstReceiveTimestamp":"1630273675966"},"messageAttributes":{},"md5OfBody":"09e622e9192285e41f52bfa16f4706a9","eventSource":"aws:sqs","eventSourceARN":"arn:aws:sqs:eu-west-1:xxxxxx:lambda-test","awsRegion":"eu-west-1"}]}
2021/08/29/[$LATEST]0d137fd7aea444e782793cc43893ea31 2021-08-29T21:47:56.089000 END RequestId: 32163cb2-4d34-55b0-ad82-65144ff0daf0
2021/08/29/[$LATEST]0d137fd7aea444e782793cc43893ea31 2021-08-29T21:47:56.089000 REPORT RequestId: 32163cb2-4d34-55b0-ad82-65144ff0daf0	Duration: 69.31 ms	Billed Duration: 70 ms	Memory Size: 128 MBMax Memory Used: 66 MB

That is it! Now, we are ready to take the advantage of AWS Lambda. You can start from here and develop your functions with the power of event-driven systems.