AWS provides two solutions for MySQL high availability, Mulit-AZ and Read Replicas. When combined, it is possible to implement synchronous replication within a region and asynchronous replication between regions using only AWS services. The two levels of protection ensure data is safe in the event of a single AZ or regional outage. It also helps in the event RDS is running smoothly but another service that an application relies on in the region is experiencing issues. A multi-region read replica provides the option of moving to a new healthy environment.
To simplify this process, which is a combination of patterns found in the Multi-AZ, Read Replica and CloudFormation documents, two CloudFormation templates were created with condensation that do all the heavy lifting. The particles are part of the condensation-examples project.
The use of condensation implements DRY principles when creating CloudFormation templates. The two templates, primary and secondary, are built from the same particle sets. This ensures each will implement an identical infrastructure in both regions. In this case, only the default parameter values are different, minimising user input in each region.
Since both the primary and secondary region share templates built from the same particles, failover is completed by taking a snapshot of the read replica and updating the secondary stack to enable Multi-AZ based on that snapshot. CloudFormation will then initialise a RDS Multi-AZ Instance alongside the read replica in the recovery region.
For more on condensation and how particles make building templates fun, take a look at our project on GitHub.
What Will Happen?
The master template takes advantage of nested stacks. When finished, each region will contain six stacks that will have built:
- 1 VPC
- 2 Subnets with routing table associations
- 2 SecurityGroups
- One for the VPC
- One for the DB Instance
- The Primary template will create an RDS Multi-AZ Instance
- The Secondary template will create a single read replica
Let’s Do It
This example will use us-east-1 and us-west-2. Any set of regions will work. In fact, condensation deploys these templates to a bucket in every AWS region (see condensation-examples for a list of buckets).
Primary Region (us-east-1)
- Switch to the us-east-1 (N.Virginia) region
- In CloudFormation create a new stack named ‘MultiRegion-RDS-Primary’ and specify the following Amazon S3 template URL: https://s3.amazonaws.com/condensation-examples.us-east-1/0.4.1/particles/cftemplates/layouts/rds/mysql_multi_az_with_read_replica_primary.template.json
- Click Next -> Next -> Create
- There is no need to change any of the default values.
- Wait for CloudFormation to finish and copy the output value for RDSARN.
Secondary Region (us-west-2)
- Switch to the us-west-2 (Oregon) region.
- In CloudFormation create a new stack named ‘MultiRegion-RDS-Secondary’ and specify the following Amazon S3 template URL: https://s3-us-west-2.amazonaws.com/condensation-examples.us-west-2/0.4.1/particles/cftemplates/layouts/rds/mysql_multi_az_with_read_replica_secondary.template.json
- Fill SourceDBInstanceIdentifier with the RDSARN output value from the primary region.
- There is no need to set any other values
- Click Next -> Next -> Create
Once the secondary stack is complete the solution is ready. The RDS Multi-AZ Instance in us-east-1 will be sending asynchronous updates to the read replica in us-west-2.
If us-east-1 experiences issues or completely fails, do the following in us-west-2:
- Create a snapshot of the read replica
- Update the CloudFormation stack
- Set CreateRDSResource to true
- Set DBSnapshotIdentifier to the Snapshot Name of the snapshot from step 1
- Use the existing value for MasterUserPassword
- Point applications at the new RDS Multi-AZ Instance in us-west-2
This is just one example of protecting and moving data between AWS regions. The intent was to use only AWS services without relying on extra compute or third party software. Within AWS, RDS has more features such as read replica promotion and chaining could be used to enhance this example. Venture outside AWS to third party tools and the possibilities to protect and make MySQL highly available grow immensely. At the end of the day it is important to pick a solution that will satisfy a Recovery Point Objective and Recovery Time Objective that is conducive to the business and the application.