By Greg Cox
When most people begin working with CloudFormation, they usually start with examples or tutorials they find online. After that, they quickly start combining and adding their own content. Very soon, if not immediately, they face the decision of nesting and how much to include in a single template, not to mention the overall scope of a group of nested stacks. The obvious goal is to create as reusable a sub-template as possible.
However, I've noticed two main pieces that most tutorials do not mention -- although everyone figures this out eventually. I hope to speed up that learning curve in this short post.
You’ve probably created or deployed a few templates and know what the parameters are, and how you pass in values to your templates that are specific to the deployment you hope to accomplish. You might start out hard coding values specific to that deployment, but if you made that value into a parameter, you would want to make that template more reusable. This is pretty obvious, though not always followed, although this is what people do to test things out. A reusable template is the goal.
One constraint of CloudFormation is the limitation in the complexity of the data you can implement within a parameter. One way around this is to standardize around an external datastore for those values and have a parameter that identifies that datastore and any sort of key for the resource you are creating in your template.
For example, for ECS you could pass on the parameter store information to allow your container to access that parameter store, and then lookup your parameters as needed within your container. You would do this as opposed to passing environment variables as parameters within your CloudFormation to your container/task definition. You will most likely always need different key/value parameters per ECS service, so altering your CloudFormation templates per service goes against having generic reusable templates.
There are many similar systems, such as etcd, consul, or any datastore like DynamoDB, but this varies depending on your comfort level. This includes your ability to run your own third-party key/value stores, possibly doing so if you are already using those system in a different part of the same environment
The key thing most people don’t realise or think about when first coming to CloudFormation is the idea of keeping everything linked to a master CloudFormation template that must be kept consistent, upgraded, and rolled back atomically. Otherwise, you will be required to add additional logic to keep things in sync – and I can almost guarantee that your environment will be out of sync. Or, if you group unrelated things in a master template, you will eventually rollback resources that don’t need to rollback because of an unrelated failure.
Again, the main idea with a master template (and nested templates linked to that master) is that if anything fails within a run, everything that needs to be kept in sync will be rolled back consistently. This is always important, but especially so when adding CloudFormation Actions to any sort of CI/CD Pipeline.
It may seem easier in the moment to run your templates separately from within your pipeline, but in the long run everything will be much cleaner if your related resources that are created/updated/deleted from your templates are kept inside of a single master template. This could also result in running into CloudFormation limits related to number of phases, stages, parallel or total actions.
As you progress with CodePipeline, at the beginning you will want to just get things working. But you will eventually find CodePipeline limits, or your pipeline just won't perform according to the goals you set for time, speed or just logical groupings. Think about how those groupings are for an efficient Development/Deployment workflow with the ability to rollback cleanly, and you will save yourself headaches in the future.
In the end, CloudFormation is all about helping you set up AWS resources so you can focus on your apps that run in AWS, vs. managing those resources. Good luck with your CloudFormation efforts; I hope I've been able to save you a little time and eliminate a little confusion in the process.