In organizations consisting of multiple teams/departments responsible for managing infrastructure on Amazon Web Services (AWS) for their respective product(s), it can be difficult to ensure that infrastructure remains exactly as it was setup. Certain AWS resources, such as Security Groups, are critical to the overall security of the product. At the time of this writing, AWS does not provide a service to notify product owners of changes to the configuration of their AWS resources. In this article, we’re going to demonstrate how to implement a system to notify product owners when changes are made to a Security Group. Although we’re going to focus on Security Groups in this article, the code samples and outlined configuration can be easily adapted to work for other types of resources.
Let’s use an example scenario to demonstrate why this functionality could be useful and important. Department A owns a Security Group that allows HTTP traffic (port 80) from the public internet (0.0.0.0/0). They want to make sure that the only Inbound Rule for this Security Group is for HTTP traffic on port 80, so they create a new Security Group that they intend to be only used by their department. Let’s now imagine that Department B comes along in need of allowing HTTP traffic to their resources. They see Department A’s Security Group and decide to use it. Department B continues designing or building their infrastructure and decide they would like this Security Group to also allow RDP connections on port 3389. (Hopefully, in a real organization there are policies and processes for making changes to Security Groups, but for illustration purposes we will continue with this scenario.) Unfortunately, they failed to coordinate with Department A and now all of Department A’s resources associated with the Security Group allow RDP access, all without any communication to Department A. Several days/weeks/months later, an operations engineer from Department A reviews the Security Group and sees the rule allowing RDP access, determines it is a vulnerability, and removes the rule. This causes connectivity issues for Department B, resulting in confusion and frustration in both departments.
If only a notification would have been sent to Department A when the Security Group was changed, there could have been less confusion, frustration, and time being vulnerable. Now that the stage is set, let’s walk through how we’re going to implement this system.
System Overview
AWS provides a service that gives us 90% of what we need to implement this system. AWS Config is a service that tracks changes to resources and can send notifications (using SNS) when changes do occur. For small organizations, using only AWS Config might suffice. However, if your organization has multiple teams or departments, being notified for every change for every department can become annoying in a hurry. You only want to be notified about your resources. What we really need is something that filters those notifications so that we only see the notifications that pertain to our department. This is a perfect situation for Lambda. Lambda functions can subscribe to an SNS topic, allowing them to receive SNS messages from AWS Config, and then filtering them so that only the changes that pertain to a department are sent.
SNS
First, we need to setup an SNS topic that will receive all the notifications that AWS Config generates. Login to the Management Console, go to the SNS page, and create a new SNS topic named something like “aws-config.”
AWS Config
Now head over to the AWS Config service page.
If you’ve already setup/enabled AWS Config, you need to verify the following:
- “EC2: SecurityGroup” is included as a resource type for which to record changes.
- Changes are being streamed to an SNS topic.
If not, let’s setup AWS Config:
- Click the “Get Started” button.
- In the input for “Select the resource types you want to record” select “EC2: SecurityGroup”.
- AWS Config uses log files to track changes to resources, and needs a location in S3 to store these log files. Choose “Create a new bucket,” unless you want the log files to be stored somewhere specifically.
- Select the check box for “Enable configuration changes and notifications to be streamed to an Amazon SNS topic,” choose a topic, and make note of selected topic.
- Click the “Continue” button
Make sure it says, “Recording is on,” and you should be set to track resource changes.
AWS Lambda
Now head over to the AWS Lambda service page. We’re going to create the function that will handle filtering the AWS Config notifications. Click the “Create a Lambda function” button, skip selecting a template, and you will be taken to the function configuration page. Give it a meaningful name (handleSGChangeNotification) and description, and make sure “Node.js” is selected as the runtime.
As we won’t need any 3rd-party dependencies, we can use the inline code editor to create our Lambda function. Copy the code provided in the Gist below:
https://gist.github.com/jordanfarrer/bca39345d410370bc821
Enter “index.handler” for the “Handler” input text box.
Next you’ll need to select a Role for Lambda to use when executing this function. Select “* Basic Execution Role” which will open a new tab to create a new Role. For “IAM Role” choose “Create a new IAM Role.” Give the role a name (such as “lambda-with-ses”). Show the policy document and click the “Edit” link. We’re using SES in this function so in addition to the base permission Lambda requires, we’ll add SES access. Copy and paste the following snippet into the policy text area:
https://gist.github.com/jordanfarrer/c83b894898553e84b452
Click the “Allow” button, and you will be returned to the Lambda function configuration page with your newly created role selected. Leave the Memory and Timeout settings as they are. Click the “Next” button to review the configuration for the function. Click “Create function.”
Now click the “Event Sources” tab. Add a new event source by clicking the “Add event source” link which will open a modal. Select “SNS” as the “Event source type.” Then select the SNS topic that AWS Config is configured to send messages to. Enable the event source, and click “Submit.” At this point, we’re ready to test.
The Lambda function looks for two tags on a Security Group when determining if/how to send a notification.
The “NotifyOnChange” tag is a switch that enables you to turn notifications on or off for this specific Security Group. Enter “Yes” if you want notifications to be sent for a specific Security Group or “No” if not.
The “OwnerDL” tag specifies to which email address notifications should be sent on a change. The value must be a valid email address.
In order to test that everything is working as intended, either create a new Security Group or find an existing Security Group. Ensure that the 2 required tags are populated. Next make a change to the Security Group such as adding or removing a rule. Within 10 minutes, you should receive an email to the email address provided in the value of the OwnerDL tag. If not, double check the steps in this article and make another change to the Security Group. If problems still persist, double check your SES settings to ensure email delivery is allowed to the email address specified, and from the email address specified in the Lambda function.
Hopefully this was a helpful demonstration to show how integrating multiple AWS services can provide a useful tool. Feel free to modify the provided code to fit your needs.
Feel free to leave feedback in the comments.
-Jordan
