The situation
Years of CloudFormation stacks, manual console changes, and undocumented fixes had produced an environment where nobody could say with confidence what existed or why. Changes were risky because the blast radius was unknown, and a single-region deployment meant a regional AWS outage was a company outage.
What I did
- Catalogued what actually existed across the accounts, and reconciled it against what the team believed existed. The gap was instructive.
- Migrated the environment to Terraform in stages, importing live resources so nothing had to be rebuilt to become code.
- Introduced ephemeral environments so every change could be tested against a real copy before touching production.
- Designed and built multi-region disaster recovery with a tested, documented failover procedure.
The outcome
The team has a version-controlled, documented foundation they run and extend themselves. An outage in one AWS region is no longer an outage for their customers. Change went from something to fear to something routine.