AWS provides two solutions for MySQL high availability, Mulit-AZ and Read Replicas. When combined, it is possible to implement synchronous replication within a region and asynchronous replication between regions using only AWS services. The two levels of protection ensure data is safe in the event of a single AZ or regional outage. It also helps in the event RDS is running smoothly but another service that an application relies on in the region is experiencing issues. A multi-region read replica provides the option of moving to a new healthy environment.
To simplify this process, which is a combination of patterns found in the Multi-AZ, Read Replica and CloudFormation documents, two CloudFormation templates were created with condensation that do all the heavy lifting. The particles are part of the condensation-examples project.
The use of condensation implements DRY principles when creating CloudFormation templates. The two templates, primary and secondary, are built from the same particle sets. This ensures each will implement an identical infrastructure in both regions. In this case, only the default parameter values are different, minimizing user input in each region.
Since both the primary and secondary region share templates built from the same particles, failover is completed by taking a snapshot of the read replica and updating the secondary stack to enable Multi-AZ based on that snapshot. CloudFormation will then initialize a RDS Multi-AZ Instance alongside the read replica in the recovery region.
For more on condensation and how particles make building templates fun, take a look at our project on GitHub.
The master template takes advantage of nested stacks. When finished, each region will contain six stacks that will have built:
This example will use us-east-1 and us-west-2. Any set of regions will work. In fact, condensation deploys these templates to a bucket in every AWS region (see condensation-examples for a list of buckets).
Once the secondary stack is complete the solution is ready. The RDS Multi-AZ Instance in us-east-1 will be sending asynchronous updates to the read replica in us-west-2.
If us-east-1 experiences issues or completely fails, do the following in us-west-2:
This is just one example of protecting and moving data between AWS regions. The intent was to use only AWS services without relying on extra compute or third party software. Within AWS, RDS has more features such as read replica promotion and chaining could be used to enhance this example. Venture outside AWS to third party tools and the possibilities to protect and make MySQL highly available grow immensely. At the end of the day it is important to pick a solution that will satisfy a Recovery Point Objective and Recovery Time Objective that is conducive to the business and the application.