I’ve been beating our drum for a while now about the inevitability of failure in cloud-based systems. Simply put, the complexities and interdependencies of the cloud make it nearly impossible to avoid service failure, so instead we have to go against our instincts and actually design for this eventuality.
Once you accept this basic premise, the next question is how exactly do we need to change our design processes? The Resilience Modeling and Analysis (RMA) methodology is a key part of the answer.
RMA brings the master carpenter’s “measure twice, cut once” philosophy to engineering. The goal is to help ensure teams think through as many of the potential reliability-related issues as possible before committing code to production—not to prevent every single failure mode, but to limit the impact a failure could have on customers if they occur.
To be clear, RMA is deeper and broader than basic fault modeling and root-cause analysis. Adapted from the industry-standard technique known as Failure Mode and Effects Analysis (FMEA), RMA is a four-phase process:
- Pre-work: Diagram your resources, dependencies, and component interactions.
- Discover: Identify potential failures and resilience gaps for each interaction identified in the pre-work phase.
- Rate: Perform an impact analysis of the potential failures you’ve identified.
- Act: Invest in and produce work items to improve resilience.
By working through these four phases, teams can gain a more detailed understanding of where known failure points are, what the impact of known failure modes is likely to be, and where to target engineering investments to help mitigate the highest-priority risks.
Feedback we’ve received from service teams who have worked through this process, is that one of the key outcomes is spending less post-deployment time managing and responding to live-site issues. Tightening the focus to reducing the impact of the most likely failures reclaims time to spend on the fun stuff—like developing customer-facing innovations.
