HOW-TO: Automating AWS Lambda Notifications using CDK and Slack

Published in

Make It New

8 min readOct 18, 2023

This article is a step-by-step tutorial on how to set up an automated metric notification system using AWS CDK. The project involves Node, TypeScript, AWS, CDK, Lambda, SNS, and CloudWatch. Even though this example will be about monitoring Lambdas, any applicable metric can be monitored this way. For the purpose of this exercise, we will push the generated notifications to Slack.

The project code is publicly available on my GitHub here.

What is AWS CDK?

AWS Cloud Development Kit (CDK) is a well-established AWS service that wraps declarative CloudFormation template syntax into an imperative programming paradigm that supports many general-purpose programming languages: TypeScript/JavaScript, Python, C#, Java, and Go.

Why?

Coming from the Serverless Framework background, I was quite used to having Lambda alarms available (almost) out of the box for each Lambda — thanks to this plugin. Lambda alarms are especially useful for catching errors early, and only at the cost of $0.10 per alarm per month. However, AWS CDK is pretty bare-bones as it has no plugins, so we will need to set this up manually. Let’s get to it!

The Plan

The architecture of the solution is very linear:

Create a dummy Lambda Function that will fail on invocation — let’s call it errorLambda ;
Create an Alarm that will monitor the error count over a period of time. Attach this alarm to the errorLambda function;
Create an SNS Topic that will receive error notifications;
Create an Alarm Action that will trigger whenever the Alarm enters the “In alarm” state. The actual action is to send a message to the SNS topic from step 3;
Create another Lambda Function that will consume SNS messages. This is the Lambda that will send notifications to Slack.

The diagram below summarises this in an ideal setup:

However, for the exercise, we will stick to having a single stack for the sake of simplicity. We will also create only one errorLambda , but any amount of Lambdas/alarms can report to the same SNS topic.

Prerequisites

You should have the following software already installed locally:

AWS CLI (requires Python 3);
AWS CDK — can be installed globally via npm install -g aws-cdk ;
Node 16+ — I will use Node 18.

The only extra NPM package that we will need is package.json is @types/aws-lambda for type definitions for Lambda inputs.

The Implementation

Let’s start by creating a dummy CDK project:

mkdir lambdaNotification
cd lambdaNotification
cdk init app --language typescript
# this command will read the name of the folder it is in, so your variables
# and files may have different names if the initial folder is called something
# other than "lambdaNotification"

This command will scaffold a generic CDK application.

NOTE: I do not recommend using this template for any serious TypeScript development due to the lack of many basic features (linting, formatting, proper bundling), but it will suffice for this exercise.

Error Lambda

Let’s define the step #1 Lambda in lib/lambda_notification-stack.ts . The only purpose of this Lambda is to error, which should trigger the notification logic. In the constructor , let’s create a new Lambda:

// lib/lambda_notification-stack.ts
// inside of the constructor
const errorLambda = new cdk.aws_lambda.Function(this, 'ErrorLambdaFunction', {
  description: 'This Lambda throws errors on invocation',
  runtime: cdk.aws_lambda.Runtime.NODEJS_18_X,
  handler: 'error.handler',
  code: cdk.aws_lambda.Code.fromAsset('lib/Lambdas/'),
});

The CDK Lambda definition points to a yet non-existent lambda handler file. Let’s create it at lib/Lambdas/error.ts :

// lib/Lambdas/error.ts
import type { Handler, Context } from "aws-lambda";

export const handler: Handler = async (_event: unknown, _context: Context) => {
  throw new Error("Oh no, an error!");
};

CloudWatch alarm

Step #2 is to create a custom alarm that can monitor Lambda errors. Any Lambda function has a set of built-in metrics in CDK that can be accessed via errorLambda.metric...() . The default error metric (errorLambda.metricErrors() ) counts the sum of errors over a 5-minute interval. For some cases that is sufficient, but I think we can do better! Let’s create a custom alarm metric that will trigger if the maximum amount of errors exceeds 0 over a 60-second interval.

Once again, we add code inside of the constructor:

// lib/lambda_notification-stack.ts
// inside of the constructor, after the lambda definition

// Define a custom Metric for the errorLambda above
const errorMetric = new cdk.aws_cloudwatch.Metric({
  metricName: "Errors", // !
  namespace: "AWS/Lambda",
  period: cdk.Duration.minutes(1),
  statistic: "max",
  dimensionsMap: {
    FunctionName: errorLambda.functionName
  }
})

! Note: metricName must match one of the allowed metric values. The options are: Errors , Throttles , Duration , Invocations , and ConcurrentExecutions .

Then, this error metric can be used for the alarm:

// lib/lambda_notification-stack.ts
// inside of the constructor, after the lambda definition

// Define a CloudWatch alarm that will monitor Lambda errors
const errorAlarm = new cdk.aws_cloudwatch.Alarm(this, 'LambdaErrorAlarm', {
  metric: errorMetric,
  // metric: errorLambda.metricErrors(), // the default alternative
  threshold: 0, // Trigger if there are more than 0 periods with errors
  evaluationPeriods: 1,
  comparisonOperator: cdk.aws_cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  alarmDescription: 'Alarm for Lambda errors',
  treatMissingData: cdk.aws_cloudwatch.TreatMissingData.NOT_BREACHING,
});

Some details:

treatMissingData is set to NOT_BREACHING . The errorMetric will report points only when the error happens; otherwise, the metric will not report any data. When this happens and there is no data, it just means that there are no errors. Thus, this behavior should be treated as not breaching the alarm (i.e. everything is good);
comparisonOperator and threshold are the alarm-defining properties. The errorMetric just shows the maximum amount of errors in 60 seconds — it is not aware of what that data means. comparisonOperator combined with threshold create a condition that, when exceeded, triggers the alarm;
evaluationPeriods defines how many individual periods must satisfy the alarm condition in order for the alarm to trigger. For example, with only 1 period, 1 error in the first 60 seconds will already trigger the alarm. But with 2 periods, 10 errors in the first 60 seconds will not trigger the alarm, whereas a single error in the first 60 seconds followed by another error in the second 60 seconds will trigger the alarm.

SNS topic

Step #3 is to create an SNS topic that will receive and send out CloudWatch error notifications:

// lib/lambda_notification-stack.ts
// inside of the constructor

// Define an SNS topic
const errorTopic = new cdk.aws_sns.Topic(this, 'ErrorAlarmListener');
// bind the alarm to the SNS topic defined above
errorAlarm.addAlarmAction(new cdk.aws_cloudwatch_actions.SnsAction(errorTopic));

Notifier Lambda

The last big step is to create a Lambda function that will be triggered by the SNS topic. Since the Lambda must be able to read messages from SNS, it will require a set of additional IAM rights. Thus, this part can be split into two steps:

Create a Lambda function and subscribe it to the SNS topic;
Expand the Lambda IAM role with additional SNS rights.

Let’s create the Lambda:

// lib/lambda_notification-stack.ts
// inside of the constructor

// Define a Lambda function that will consume the SNS messages
const notifierLambda = new cdk.aws_lambda.Function(this, 'NotifierLambdaFunction', {
  description: "This Lambda consumes SNS alarm messages",
  runtime: cdk.aws_lambda.Runtime.NODEJS_18_X,
  handler: 'notifier.handler',
  code: cdk.aws_lambda.Code.fromAsset('lib/Lambdas/'),
  events: [new cdk.aws_lambda_event_sources.SnsEventSource(errorTopic)],
});

We reference the previously created errorTopic as the input event. For now, the Lambda handler at lib/Lambdas/notifier.ts can just log the received SNS event:

// lib/Lambdas/notifier.ts

import type { Handler, Context, SNSEvent } from "aws-lambda";

export const handler: Handler = async (event: SNSEvent, _context: Context) => {
  console.log(JSON.stringify(event, null, 2));
  return {
    result: "Done!",
  }
};

Let’s extend the IAM role:

// lib/lambda_notification-stack.ts
// inside of the constructor

// Attach a policy to the lambda role granting permissions to read from the SNS topic
const snsPolicy = new cdk.aws_iam.PolicyStatement({
  actions: ['sns:Subscribe', 'sns:Receive'],
  resources: [errorTopic.topicArn],
}); // 1
notifierLambda.addToRolePolicy(snsPolicy); // 2

Notes:

Create an IAM policy with SNS-specific permissions: topic subscription (sns:Subscribe) and message receival (sns:Receive);
Add the policy to the existing Lambda IAM role.

Testing

Now let’s test and deploy!

Validate that the underlying CloudFormation template compiles successfully via npm run cdk synth in the project root;
Deploy via npm run cdk deploy in the project root;
Manually trigger the errorLambda function once the stack is deployed (for example, via GUI’s Testbutton). Within the next 60 seconds, the errorAlarm should trigger and the notifierLambda should receive an SNS message. You can check it by going into the CloudWatch logs for the notifierLambda :

Technically, the tutorial is done at this stage. If you are not interested in Slack integration — jump over to the conclusion.

Adding Slack Integration

First, you need to create a Slack App here. Follow this tutorial on how to do that. If done correctly, you should have this Webhook URL available on the App > Features > Incoming Webhooks page:

Now we are ready to add this to our Lambda. Let’s modify notifier.ts :

// lib/Lambdas/notifier.ts

import type { Handler, Context, SNSEvent } from "aws-lambda";

const url = ""; // Paste your URL here
if (!url) throw new Error("No Slack URL present!");

export const handler: Handler = async (event: SNSEvent, _context: Context) => {
  console.log(JSON.stringify(event, null, 2));

  // 1
  const record = JSON.parse(event.Records[0].Sns.Message);
  const functionName = record.Trigger.Dimensions[0].value;
  const alarmDescription = record.AlarmDescription;
  const time = record.StateChangeTime;

  const alarmArn = record.AlarmArn;
  const region = alarmArn.split(":")[3];

  const cloudwatchLink = `https://${region}.console.aws.amazon.com/cloudwatch/home?region=${region}#logsV2:log-groups/log-group/$252Faws$252Flambda$252F${functionName}`;

  // 2
  const data = {
    "blocks": [
      {
        "type": "section",
        "text": {
          "type": "mrkdwn",
          "text": `:exclamation::exclamation::exclamation:*Lambda error*:exclamation::exclamation::exclamation:\n${functionName}`
        }
      },
      {
        "type": "section",
        "text": {
          "type": "mrkdwn",
          "text": `${alarmDescription}`
        }
      },
      {
        "type": "section",
        "fields": [
          {
            "type": "mrkdwn",
            "text": `*Time*\n${time}`
          },
          {
            "type": "mrkdwn",
            "text": `*Region*\n${region}`
          }
        ]
      },
      {
        "type": "section",
        "text": {
          "type": "mrkdwn",
          "text": `<${cloudwatchLink}|CloudWatch logs>`
        },
      },
    ]
  }

  // 3
  await fetch(url, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify(data)
  })

  return {
    result: "Done!",
  }
};

The new handler does a couple of things:

It parses the incoming SNS message. Check the execution logs of the previous invocation of notifierLambda to see the exact JSON;
It composes a nice-looking message using Slack’s Block Kit;
It makes a POST request to the Webhook URL.

As a result, you should see the following in the configured Slack channel:

Conclusion

This tutorial has demonstrated how to build a simple CDK CloudFormation stack capable of notifying third-party applications. The Slack endpoint can be easily swapped out for any other integration. The same applies to the errorLambda function — it can be swapped out for any other AWS component that has CloudWatch metrics available. This simple but universal setup should allow you to track the status of your cloud apps.

What’s next?

Here are some of the quality-of-life improvements that can be added to make the notifications a bit nicer:

Add an interactive button to the notification message connected to an API Gateway that can trigger a response action;
In the notifier Lambda, fetch the logs associated with the error;
Look into AWS ChatBot if you want an out-of-the-box alternative.

But this is for you to discover :)

Albert Asratyan @ Netlight Stockholm for MakeItNew

Check out my other articles here.