Lambda Formation: Rocket fuel for AWS CloudFormation

Since the launch announcement in late 2014, Lambda has become one of the most popular and fastest growing services on AWS.  With no infrastructure to worry about, there’s no question why developers have flocked to the service that will execute javascript, java or python code within milliseconds of a request.

The fast adoption of developers mirrors the speed at which AWS is integrating Lambda into the entire ecosystem.  Design patterns that once relied on a combination of SNS, SQS and EC2 are being replaced with snippets of code. By linking services to Lambda, AWS puts the power in the hands of the developers to integrate with any resource they desire. Manipulate and transform data, call other AWS services with the corresponding AWS SDK, or fire off API calls to third party services.  It is becoming clear that Lambda is the glue that is bringing exciting integrations to the building blocks of AWS.

As the author of condensation one of the integrations I find most interesting involves CloudFormation.  CloudFormation is the best tool for declaring an AWS environment.  It provides a clean way to define, change and ultimately destroy stacks of resources. The goal is to never manually provision anything, ever.  Unfortunately, CloudFormation does not always have the latest integrations to new AWS services.  Sometimes, there are even some older, more mature services that have an odd use case or less often used component that also fail to become a resource definition.

Before Lambda, there would be some fist shaking, possibly some harsh words, maybe even a tear.  The thought of using SNS would enter the brain, only to be squashed by the thought of all the other infrastructure to be implemented to make it work.  A SQS Queue, an EC2 instance, possibly a Redis store to make sure you don’t run a duplicated message twice.

Then, something even worse might happen; the resource you want to create is not within AWS.  It is an external service, a SasS, another IaaS or even a traditional datacenter.  All the same questions get re-asked again, but this time there is also a question as to where the extra compute should reside.

But then it happened. CloudFormation updated Custom Resources to call Lambda functions directly.  DevOps rejoiced.  No extra topics, queues and compute.  Just code, baby.  Pure uncut, straight from the source, code.

With this integration, CloudFormation can now create any resource with a few lines of Node.js.  The entire AWS SDK was handed over, daring anyone to complain about a slow to implement CloudFormation feature again.  On top of that, third-party APIs were now only a simple request away.  There is no longer a decision on where the integration point should live for Custom Resources.  Lambda is the answer.

As the Sungard AS labs team used this feature more and more we found some patterns repeating themselves over and over again.  There was a common structure, naming convention and set of checks that we implemented and iterated through each time Lambda and CloudFormation were used together.  As good developers do, we brought these patterns together and decided to share them with the world.  I introduce lambda-formation.

Lambda Formation

The lambda-formation project is a small, slightly opinionated way to build a set of lambda handlers that are compatible with CloudFormation Custom Resources.  The generator will drop down a skeleton to support any number of resources that can be created, updated and deleted with a CloudFormation Stack.  The guess work of where to put code and how to respond to both the Lambda service and CloudFormation is removed.  Basic sanity checks on request type and event data are provided for your safety.

Furthemore, lambda-formation projects are self-sufficient.  They are compatible with but do not rely on CloudFormation.  Any lambda-formation project can be called directly through the Lambda API and it will process and respond back to Lambda accordingly.  This flexibility keeps code DRY allowing handlers to be called by other scripts, APIs or CloudFormation.

Flight Path: A quick HowTo

We created lambda-formation-example-resources to demonstrate how lambda-formation works.  The examples show the structure created by the Yeoman generator, where can be added and how CloudFormation can be used to call each resource.

In the lib/resources/onPremFwIngressRule example, handlers mimic what calling a resource in an on-premises datacenter might look like.  The code in the example does not actually make any extra API calls, but it does show how easy it is to hook up logic to create, update and delete actions.

For details on how resources and actions can be called via Lambd or CloudFormation see the README.md for lambda-formation.

If you would like to run the example yourself, condensation was used to build particles-lambda-formation-example-resources which deploys a ready to use CloudFormation stack to all Lambda available regions.

Click one of the launch buttons below to run your own stack:

Region	Stack
us-east-1
us-west-2
eu-west-1
ap-northeast-1

Blast Off: Next Steps

Combining Lambda and CloudFormation with the ability to provision custom resources expands configuration as code to unique hybrid solutions.  Automatable, repeatable deployments are the prize and lambda-formation helps us achieve that goal.

With lambda-formation we hope to encourage the community to share new integrations for CloudFormation.  Whether those integrations are for other AWS resources or third-party SaaS and IaaS providers.  This new project  is also helping us add new Lambda features for the Sungard AS condensation project.

It should be mentioned that lambda-formation is the result of taking our final design for spotinst-lambda and making that structure work for every project.  The spotinst-lambda project will be updated to use lambda-formation in the near future and we will talk about spot instances soon too!

Please take a look at our Open Source GitHub projects and let us know how you are using AWS and Lambda today.