You're dealing with service disruptions in multiple cloud platforms. How do you decide what to tackle first?

When dealing with service disruptions across multiple cloud platforms, prioritizing what to tackle first requires a methodical approach to minimize downtime and impact. Start by assessing the severity and scope of each disruption. Prioritize issues based on their impact on critical business operations and the number of users affected. Implement an incident response plan that includes clear communication with stakeholders about the status and expected resolution time. Utilize automated monitoring tools to quickly identify and diagnose the root causes. This strategic prioritization ensures that resources are allocated efficiently, addressing the most critical disruptions first to restore services swiftly and effectively.

As a Site Reliability Engineer, I'm expected to maintain 99% uptime on a monthly basis. Every disruption is one too many as customers can be very unforgiving and churn to show their distaste. To assess the impact, we have a pro-active plan where we already configured cloud trail to inspect logs and once there is a failure, I get predefined codes via pagerduty alerts if I'm on-call. These codes already exist in our internal documentation and makes troubleshooting faster as I can easily assess the cause, estimated time of resolution, impact as well as methods of resolution. Metrics instantly lead me to quantifying data which primary determines the magnitude of the impact of the disruption.

Whichever service is the most 'disruptive' = costs the most money. Get back online as fast as you can. Review why it happened. Consider alternative vendors if disruptions are recurring and short term migration pain enables long term uptime.

At a tech startup, we faced a critical disruption in our payment processing system, impacting a large portion of our user base. We swiftly evaluated the impact based on user feedback, financial implications, and service criticality. Recognizing the revenue impact, we prioritized restoring the payment system over less critical issues. This strategic assessment enabled us to mitigate customer dissatisfaction and financial losses effectively, highlighting the importance of prioritizing disruptions based on their operational impact.

Impact assessment should always be the first step. If you don't know which system is most critical to the organization, you won't know which one to fix first.

As part of our subscription agreements, we share Service Level Agreements with our customers. Since we do not have control of our cloud vendor(AWS) we have to the same SLA from them. We then make sure we respond to incidents or support tickets in compliance with these SLAs. When we fail, we compensate our customers accordingly. I rely primarily on Solarwinds and Kingdom to measure fulfilments of SLAs but there are other alternatives like ServiceNow and Nagios

In addition to assessing the impact, the SLA with your customers and between your cloud provider and your organization is another variable to consider. I will prioritize resolving the service that generates more penalties with my customers.

Assess how disruptions align with the terms of your cloud providers' Service Level Agreements (SLAs). Evaluate the response times, escalation procedures, and compensation mechanisms outlined in the SLAs. Prioritize disruptions that exceed the agreed-upon response times or breach service thresholds, as these could lead to financial adjustments or improvements in service. Additionally, review the provider's historical performance to gauge their reliability and potential for future issues.

Are services with SLA and services without SLA? In case of a disruption, services with SLA take the highest priority. Then, according to the SLA, we can create a plan based on the SLA to set up priorities (disruptions in those services with high SLA will take the highest priority).

I update our status page with Status.io to make sure that our customers are aware of downtimes as soon as they occur and are informed according once service is restored. In status.io, I already have a set of templates indicating the kind of disruption, estimated time to resolve and apologies for the inconvenience. This saves a ton of time especially when time is most crucial. There are also broadcasts that are sent during scheduled maintenances and after post-mortems.

The communication plan is something you need to define, understand, and communicate within your organization, it is critical to know who is the communication channel with the cloud provider and who is the communication channel with our internal and external customers.

Create a comprehensive communication strategy to address cloud service disruptions. Prioritize real-time updates for key stakeholders such as clients and internal teams, using automated notifications and status dashboards. Implement a protocol for escalating critical issues and providing regular status reports on remediation efforts. This approach ensures stakeholders are informed and reassured, helping to manage the impact of outages and maintain operational continuity.

Many resolutions are not straightforward. As a result, we sometimes roll-back to a previous update while we fix the underlying issue. This implies that we already have backups in place and several mitigation steps. Things that help fix highly complex disruptions: Teamwork: Two Site reliability engineers are better than one. Proper Documentation: This is proactive documentation of already known issues an how to fix them Subject matter Expertise of stack: Knowing the stack helps a ton and you're more likely to understand the cause and fix it on time.

It's better to have at least one person on-call. I believe two is even better and have a proper way of alerting them. I rely on pager duty and this generates slack alerts, phone calls, emails and a notification on the mobile app. Even if you're not at alert, thanks to Pagerduty, you will be. We also have an on-call rotation and everyone already knows what time they have to be available to avoid prolonged disruptions.

Evaluate the technical complexity of each disruption, considering factors such as required resources and potential downtime. Address issues that can be resolved with automated tools or scripts to quickly restore functionality. For disruptions needing significant engineering efforts, implement temporary workarounds or leverage cloud provider support for expedited resolution. Prioritize tasks based on their impact on service stability and user experience.

The title of this article is about "service disruptions in multiple cloud platforms." The chances of multiple cloud platforms (like AWS, Azure, OCI, and GCP) simultaneously experiencing service disruptions is very low. The chance of multiple cloud platforms being used by your workload (in a multi-cloud environment) all having disruptions in the regions and services that simultaneously impact your workload is astronomically small.