In the space of 48 hours last week, two very different failures exposed the same truth: most organisations depend on technology they neither own nor fully understand.
On Tuesday 20 October, a power event in one Google Cloud data centre in London knocked UK banking apps, the NHS, courts and couriers offline for a morning. The same evening, details leaked of two critical flaws in PHP, the programming language that quietly runs a huge share of the world’s websites. Within hours, attackers were scanning the internet for vulnerable servers.
One incident is about over-reliance on big infrastructure. The other is about the nuts and bolts that sit underneath almost everything. Both are business continuity problems first and technology problems second. And both reward organisations that have planned, rehearsed and challenged their response before the day it is needed.
Failure one: when a single building takes down a nation’s banking apps
The trouble started shortly after 7am UK time on 20 October. Google attributed it to a power event in europe-west2-b, one of three availability zones that make up its London cloud region. Errors and intermittent network failures then spread across the wider London region, hitting services such as Cloud Run, Kubernetes Engine and App Engine.
The list of those affected reads like a cross-section of daily UK life:
- Banking: Revolut, Lloyds, Halifax, Bank of Scotland and HSBC customers reported problems. Revolut confirmed disruption to card payments, ATM withdrawals and transfers.
- Public services: parts of NHS England and the video system used by courts in England and Wales.
- Everyday services: Royal Mail, Rightmove, Spotify and the budgeting app Emma, which was down for several hours.
Google reported full service restored by around midday, and Number 10 confirmed it had been in contact with the company. Affected firms were quick to stress that customer money and data were safe, and that this was a technical failure rather than a cyber attack. That reassurance mattered. In the first hour of an outage, customers and journalists tend to assume the worst.
Notice too that the impact was not confined to the UK. Apps in other countries that happened to run on London-based infrastructure went down with it.
The lesson: big is not the same as resilient
Moving to a hyperscale cloud provider is usually a sound decision. Their engineering is excellent and their uptime is better than most in-house data centres could manage. But the outsourcing of infrastructure is not the outsourcing of accountability. When your app stops working, your customers call you, not Google.
Three uncomfortable points for continuity and crisis managers:
- Concentration risk is real. When many organisations sit on the same provider, in the same region, a single fault becomes a sector-wide event. Your competitors, your suppliers and your payment provider may all fall over at the same moment.
- “Multi-zone” may not mean what you think. Cloud regions are designed so that one zone can fail without taking the others down. This time, a problem in one zone still produced errors across the region. Plans that assume the provider’s architecture will absorb a failure need testing against the day it does not.
- Your third parties’ cloud is your cloud. You may not use Google Cloud at all, yet still have been hit through a bank, a payroll provider, a courier or a SaaS tool that does. Few business impact analyses map dependencies that far down the chain.
For regulated UK financial firms, this is not abstract. The operational resilience rules require firms to identify their important business services, set impact tolerances, and prove they can stay within them during severe but plausible disruption. A morning-long outage of a major cloud region is exactly that kind of scenario.
Failure two: a crack in the foundations
PHP is not a household name, but it powers a vast number of websites, including most WordPress sites and many customer portals, intranets and back-office tools. On 20 October, the PHP project announced an upcoming security release for two critical vulnerabilities, since nicknamed “PHP2Shell”.
In plain English: an attacker with no username or password could send a crafted request to a website and take control of the server behind it. Both flaws scored 9.2 out of 10 for severity.
What turned this from a routine patch into a crisis was the timeline:
| When | What happened |
|---|---|
| 17 October | Amazon logged an exploit attempt, four days before public disclosure |
| 20 October | PHP announced the coming fix; technical details leaked on social media the same day |
| 21 October | PHP released fixed versions. Attacks began within the hour |
| 22 October | Cloudflare reported attempts had grown from tens of thousands to millions |
| 23 October | PHP disclosed a third, lower-risk flaw with no immediate fix, only a workaround |
| 26 October | The US agency CISA added both flaws to its list of vulnerabilities known to be exploited |
Palo Alto Networks estimated more than 22 million internet-facing systems were potentially exposed. Major vendors including Cisco then had to issue their own advisories, because PHP sits inside products their customers had never thought of as “PHP systems”.
The lesson: you cannot defend what you cannot see
If the Google outage was about putting too many eggs in one basket, PHP2Shell was about not knowing which eggs you had. Ask most leadership teams whether they run PHP and you will get a blank look. Yet it is very likely somewhere in their estate: the corporate website, a marketing microsite, a supplier portal, or embedded inside a security appliance.
For continuity and crisis managers, the PHP episode raises four practical questions:
- Do we have an accurate inventory? Patching starts with knowing where the vulnerable software lives, including systems run by agencies and suppliers on your behalf.
- How fast can we really patch? Attacks began within an hour of the fix. A monthly patch cycle, or a change board that meets on Thursdays, is no longer a plan.
- Who decides to pull the plug? Some organisations chose to take services offline until they were patched. That is a business decision with customer, revenue and reputational consequences, and it needs a named decision-maker and pre-agreed criteria.
- What if there is no patch yet? The third flaw arrived with only a workaround. Crisis plans need a path for living with known exposure, not just for fixing it.
The early leak is also a reminder that the timeline of a crisis is rarely yours to control. Plans built around an orderly disclosure and a calm patch window can be overtaken by a social media post.
What the two failures have in common
One was an accident at the top of the technology stack; the other a weakness at the bottom. But from a crisis management perspective they share the same DNA.
- The cause sat outside the organisation. Neither the banks nor the website owners did anything wrong to trigger these events. They still owned the consequences.
- The first question was “Is this an attack?” In the outage, firms had to reassure customers it was not. In the PHP case, the honest answer for many was “we don’t know yet”. Both demand fast, accurate and calm communication under uncertainty.
- Decisions came before information. Leaders had to choose whether to switch to fallbacks, take sites offline or brief regulators long before they had the full picture.
- Everyone was hit at once. Shared dependencies mean shared crises. Suppliers are busy, support lines are jammed and the media is looking for the worst-affected example.
This is why technology resilience cannot be left to IT alone. The technical fix is only one strand. The others are customer communications, regulatory notification, staff briefings, supplier management and executive decision-making. Those are the strands that business continuity and crisis teams own.
Ten questions to put to your plan this week
Sit down with your crisis team and ask:
- Which of our important business services depend on a single cloud provider or region, directly or through a supplier?
- If that region went dark at 7am tomorrow, what would customers experience in the first hour?
- Do we have a tested fallback, or just a documented one?
- How long can each critical service be down before we breach our impact tolerance?
- Do we know where PHP, or any other foundational software, runs across our estate and our suppliers’?
- How quickly could we patch an internet-facing system if attacks started within the hour?
- Who has the authority to take a customer-facing service offline, and on what criteria?
- Do we have holding statements ready for “technical outage, not a cyber attack” and for “we are investigating a potential compromise”?
- When would we notify regulators, and who drafts that message?
- When did we last exercise a third-party technology failure, rather than one we caused ourselves?
Plans are only as good as the last time you tested them
The organisations that coped best last week were not the ones with the thickest plans. They were the ones whose people had already practised making decisions with incomplete information, under time pressure, with the phones ringing.
That is what Crisis Cloud AI is built for. It is a self-service crisis exercise platform that lets you upload your plan. The AI analyses the plan and uses this to design an exercise, hosts the exercise with simulated news, social media and messaging and then compares participant messaging to expected plan behaviours
For operational resilience testing, you can build exercises around your important business services and their maximum tolerable disruption, so you see exactly where a scenario would push you past your limits.
The result is a rehearsal that feels like the real thing, and a clear report detailing how your plan fared. Crisis Cloud AI lets you play out the next crisis before it arrives.
Want to see how your plan would cope? Click here for a no-signup demo.