How to deploy CloudFormation

In this guide we will demonstrate how to use the AttiniCfn step to deploy a CloudFormation stack from our deployment plan.

Introduction

CloudFormation is AWS own Infrastructure as Code (IaC) language, and it’s a good tool for creating different Cloud resources. It’s also the underlying technology for other tools, like AWS CDK and AWS SAM. Although creating resources in the web console is easy, it quickly becomes hard to manage. Therefore, it is highly recommended to use IaC instead.

Attini has a custom step type for deploying CloudFormation, the AttiniCfn step. It is easy to use and comes with a few noteworthy features:

Event driven and fast

Because the deployment plan runs within your AWS account, it can be event-driven when deploying Cloudformation instead of using a polling pattern. The problem with using a polling pattern is that it makes the pipeline slow, especially if there is no change to a stack. A step can never be faster than the polling interval, meaning that a big pipeline with a lot of steps that require no action becomes needlessly slow.

Cross-Region and Account deployment

Although not the most common use case, sometimes we need to deploy a stack to a different AWS Account or Region. This is useful if we want to centrally manage resources in multiple accounts, for example AWS Config rules. Another common scenario is CloudFront certificates, which must be created in the us-east-1 region.

Configuration

CloudFormation stacks often need a lot of configuration and we normally want to keep the configuration separate from the template. With the AttiniCfn step you can write configuration files in either JSON or YAML. They also support inheritance and variable substitution, making them very flexible.

How to use

In its most minimal format the AttiniCfn step looks like this:

Type: AttiniCfn
Properties:
  StackName: String
  Template: String

The “StackName” field specifies the name of the stack and the “Template” field specifies a path to the CloudFormation template. The template path can either be an S3 Path, starting with “s3://”, or a path to a file in the distribution, starting with “/”. Although these fields are mandatory, they don’t have to be specified inline. You can also put them in a configuration file, meaning that the following is also valid:

Type: AttiniCfn
Properties:
  ConfigFile: /config.yaml

Configuration file:

stackName: String
template: String

All properties that can be set inline can also be set via a configuration file (except for the “ConfigFile” field, for obvious reasons).

There are a lot of different properties that can be set, for a complete list please refer to the documentation. But the most common one is probably the “Parameters” field which is used for passing parameter values to a stack. So if a template has a parameter called “MyParameter” then the step could look like this:

Type: AttiniCfn
Properties:
  StackName: MyStack
  Template: /template.yaml
  Parameters:
    MyParameter: my-value

The output from a stack will be passed to the following steps in the deployment plan via the payload. This removes the need for CloudFormation exports, which creates unnecessarily stale dependencies between stacks.

Demo

For this demo we will create a Distribution with three steps. The first step will create a DynamoDB table. The second step will deploy a Lambda that will return the number of times it has been called. It will do this by incrementing a counter with 1 each time it is called and then returning the counter. To persist the counter it will use the DynamoDB table we created in step one. Lastly we create a third step that simply calls the Lambda. The result is a distribution that counts how many times it has been deployed in the given region and account.

Our Distribution has the following structure:

.
├── attini-config.yaml
├── database.yaml
├── deployment-plan.yaml
└── lambda.yaml

And our deployment plan looks like this:

DeployCfnDemo:
  Type: Attini::Deploy::DeploymentPlan
  Properties:
    DeploymentPlan:
      - Name: Deploy_CounterDatabase
        Type: AttiniCfn
        Properties:
          StackName: !Sub ${AttiniEnvironmentName}-counter-database
          Template: /database.yaml
      - Name: Deploy_CounterLambda
        Type: AttiniCfn
        Properties:
          StackName: !Sub ${AttiniEnvironmentName}-counter-lambda
          Template: /lambda.yaml
          Parameters:
          DatabaseName.$: $.output.Deploy_CounterDatabase.DatabaseName
      - Name: Invoke_CounterLambda
        Type: AttiniLambdaInvoke
        Parameters:
          FunctionName.$: $.output.Deploy_CounterLambda.FunctionName

How to write CloudFormation is a bit out of scope for this demo, but you can find all the templates at the end of this page.

For our Lambda to know which DynamoDB table to use we need to pass it the table name. So in the database template, we make sure to include it in the output, for example:

Outputs:
  TableName:
    Description: "Database table name"
    Value: !Ref Database

We can then read the value from the deployment plan payload and pass it to the Lambda template as a parameter. The output from a step is always placed under the steps name in the output section of the payload.

Let’s deploy by running the deploy run command from the root of the project:

attini deploy run .
example of deploying cfn

We can see that the final step that invoked our Lambda returned 1 because it was the first time we ran it. So everything appears to work as intended.

Adding some configuration

Let’s say that we want to add some additional configurations for our stacks. We can do this by writing a configuration file. It’s good practice to keep configuration in separate files for several reasons:

  1. It reduces clutter in our deployment plan.
  2. It makes configuration reusable between steps.
  3. We can use configuration features that are not available for inline configuration, like inheritance.
  4. It makes it easier to keep different configurations for different environments.
  5. Changes to the configuration file will not trigger a change in the init stack, reducing the number of times we need to update it and therefore making our deployments faster.

So let’s create a configuration file for our Lambda called, lambda-config.yaml.

parameters:
  MemorySize: 256

Then we update the Lambda step to look like this:

- Name: Deploy_CounterLambda
  Type: AttiniCfn
  Properties:
    StackName: !Sub ${AttiniEnvironmentName}-counter-lambda
    Template: /lambda.yaml
    ConfigFile: /lambda-config.yaml
    Parameters:
      TableName.$: $.output.Deploy_CounterDatabase.TableName

We can keep the TableName parameter inline if we want. The parameters from the configuration file will be merged with the inline parameters, with the inline parameters always taking precedence.

Let’s redeploy the distribution with the new configuration!

example of deploying cfn

Notice that the database step was near instant this time because there were no updates to the stack.

AWS state language does not work in configuration files, meaning that we can’t directly read from the payload. However if we want to use a value from the payload in a configuration file, we can use variables. So we can change the step to look like this:

- Name: Deploy_CounterLambda
  Type: AttiniCfn
  Properties:
    StackName: !Sub ${AttiniEnvironmentName}-counter-lambda
    Template: /lambda.yaml
    ConfigFile: /lambda-config.yaml
    Variables:
      TableNameVariable.$: $.output.Deploy_CounterDatabase.TableName

We can then use variable substitution in the configuration file:

parameters:
 MemorySize: 256
 TableName: ${TableNameVariable}

Configuration files also support inheritance. So we can create a parent configuration that is shared by many steps and then write more specialized files for our different steps. Let’s create a parent configuration containing a tag that we should use for both our database stack and our Lambda stack. We call it parent-config.yaml.

tags:
 Author: The Machine

We then update our Lambda configuration file to look like this:

extends: /parent-config.yaml
parameters:
  MemorySize: 256
  TableName: ${TableNameVariable}

And update our database step to use the parent configuration file:

- Name: Deploy_CounterDatabase
  Type: AttiniCfn
  Properties:
   StackName: !Sub ${AttiniEnvironmentName}-counter-database
   Template: /database.yaml
   ConfigFile: parent-config.yaml

And now both our stacks will get the “Author” tag when we redeploy the distribution. Our distribution should now have the following file structure:

.
├── attini-config.yaml
├── database.yaml
├── deployment-plan.yaml
├── lambda-config.yaml
├── lambda.yaml
└── parent-config.yaml

What if we want to use different configuration files for different environments? Because the deployment plan is defined in a CloudFormation stack, we can use CloudFormation intrinsic functions. We already did this for the “StackName” field in our examples. So we could write something like this to get different configurations depending on the environment name:

ConfigFile: !Sub /${AttiniEnvironmentName}/lambda-config.yaml

The example above would read the configuration file from a folder in the distribution with the same name as the environment. For example, if the environment name is “dev” then the result would be:

ConfigFile: /dev/lambda-config.yaml

We have now gone over the most basic features of the AttiniCfn step. However, there are more features to explore. For a complete list please refer to the documentation. For more information about the step we used to invoke the Lambda, there is a guide similar to this one here.

You can also find the complete demo on Github.

Example files

Deployment-plan.yaml
AWSTemplateFormatVersion: "2010-09-09"
Transform:
 - AttiniDeploymentPlan
 - AWS::Serverless-2016-10-31

Parameters:
  AttiniEnvironmentName:
    Type: String

Resources:
  DeployCfnDemo:
    Type: Attini::Deploy::DeploymentPlan
    Properties:
    DeploymentPlan:

      - Name: Deploy_CounterDatabase
        Type: AttiniCfn
        Properties:
        StackName: !Sub ${AttiniEnvironmentName}-counter-database
        Template: /database.yaml
        ConfigFile: /parent-config.yaml

      - Name: Deploy_CounterLambda
        Type: AttiniCfn
        Properties:
        StackName: !Sub ${AttiniEnvironmentName}-counter-lambda
        Template: /lambda.yaml
        ConfigFile: /lambda-config.yaml
        Variables:
          TableNameVariable.$: $.output.Deploy_CounterDatabase.TableName

      - Name: Invoke_CounterLambda
        Type: AttiniLambdaInvoke
        Parameters:
        FunctionName.$: $.output.Deploy_CounterLambda.FunctionName
database.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: >
    Contains a simple database

Resources:
  Database:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: Id
          AttributeType: S
      BillingMode: PAY_PER_REQUEST
      KeySchema:
        - AttributeName: Id
          KeyType: HASH

Outputs:
  TableName:
    Description: "Database table name"
    Value: !Ref Database
lambda.yaml
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31

Parameters:
  AttiniEnvironmentName:
    Type: String
  TableName:
    Type: String
  MemorySize:
    Type: String
Resources:

  HelloWorldLambda:
    Type: AWS::Serverless::Function
    Properties:
      Description: !Sub Lambda that returns hello ${AttiniEnvironmentName} world
      FunctionName: !Sub ${AttiniEnvironmentName}-counter-function
      MemorySize: !Ref MemorySize
      Handler: index.lambda_handler
      Runtime: python3.9
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref TableName
      Environment:
        Variables:
          TABLE_NAME: !Ref TableName
      InlineCode: |
          import os
          import boto3

          dynamodb_client = boto3.client("dynamodb")

          def lambda_handler(event, context):
              counterItem = dynamodb_client.get_item(
                 TableName=os.environ["TABLE_NAME"],
                 Key={
                   'Id': {
                       'S': 'counter-id'
                   }
                 }
              )

              counter = int(counterItem["Item"]["counter"]["N"]) if "Item" in counterItem else 0
              counter += 1

              dynamodb_client.update_item(
                 TableName=os.environ["TABLE_NAME"],
                 Key={
                   'Id': {
                       'S': 'counter-id'
                   }
                 },
                 AttributeUpdates={
                   'counter': {
                       'Value': {
                           'N': str(counter)
                       }
                   }
                 },
              )

              return f"I have been invoked {counter} times!"          




  HelloWorldLambdaLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/lambda/${HelloWorldLambda}
      RetentionInDays: 30


Outputs:
  FunctionName:
    Value: !Ref HelloWorldLambda
lambda-config.yaml
extends: /parent-config.yaml
parameters:
  MemorySize: 512
  TableName: ${TableNameVariable}
parent-config.yaml
tags:
  Author: The Machine
attini-config.yaml
distributionName: invoke-lambda-demo
initDeployConfig:
  template: deployment-plan.yaml
  stackName: ${environment}-${distributionName}-deployment-plan

package:
  prePackage:
    commands:
    - attini configure set-dist-id --random