For anybody wondering why Microsoft keep ending up in the frame, they had an Azure outage and- this may be news to some people- a lot of Microsoft support staff are actually external vendors, eg TCS, Mindtree, Accenture etc.

Some of those vendors use Crowdstrike, and so those support staff have no systems.

But MS isn’t the outage cause today.

BBC News at 6 is leading the entire show with this. (They asked me to appear but I was slightly busy).

For the record I spent much of the day trying to tell people it isn’t a Microsoft issue.

Billboards in Times Square blue screen of deathing. Nice way to find out which orgs use Crowdstrike, this 🤣

Source is BBC News, if anybody wondering.

CrowdStrike have effectively a mini root cause analysis out

Pretty much as everybody knows, they did a channel update and it caused the driver to crash.

If they blame the person who did the update.. they shouldn’t, as it sounds like an engine defect.

crowdstrike.com/blog/technical…

Technical Details: Falcon Update for Windows Hosts | CrowdStrike

Learn more about the technical details around the Falcon update for Windows hosts.

^CrowdStrike

For the people thinking ‘shouldn’t testing catch this?’, the answer is yes. Clearly something went wrong.

This isn’t CrowdStrike’s first rodeo on this, although it is the most severe incident so far.

Eg just last month they had an issue where a content update pushed CPU to 100% on one core: thestack.technology/crowdstrik…

Truthfully these issues happen across all vendors - I’ve had my orgs totalled twice now by AV vendors, one while I was on holiday abroad and had to suspend said holiday.

CrowdStrike bug maxes out 100% of CPU, requires Windows reboots

"Note: This is 100% of a single core. In an 8-core system for example, an additional 12.5% of unexpected total CPU load would be experienced..."

^{The Stack}

Btw, that isn’t to excuse it or any vendor. CrowdStrike have gotta be better at this stuff. And they’ll have to, as if they aren’t transparent customers will flee.

It’s a warning shot to all AV/EDR/XDR vendors that if you fuck up availability, your brand will become failure. It’s harsh but that’s the media cycle and modern world.

The Verge has a quick look at the orgs trying to recover from the Crowdstrike incident.

If you’re wondering why it’s dropped off the radar of most press, they think it’s over as Down Detector looks okay (which, to be clear, is not good logic).

theverge.com/2024/7/21/2420296…

CrowdStrike outage: Photos, videos, and tales of IT workers fixing BSODs

Images, video, and stories illustrate how the CrowdStrike outage is affecting IT workers as they push to get their organizations’ systems back online.

^{Wes Davis (The Verge)}

Crowdstrike are touting auto remediation of blue screen as an opt in feature.

However, I just tried it - it’s not very successful, most boots still blue screen of death. I think CS need to be careful on messaging about this as it sounds like they’re offering it as a silver bullet. It only works if networking kicks in and the agent updates before Windows finishes booting.

reddit.com/r/sysadmin/comments…

CrowdStrike have published a video on YouTube about how to remediate PCs: youtube.com/watch?v=Bn5eRUaMZX…

(Despite the name, Self-Remediation, it is manual).

Upguard have published a list of companies they say are impacted by the CrowdStrike 'Global IT Outage', based on public reporting.

upguard.com/crowdstrike-outage

Edit: obviously it’s missing most companies as most companies aren’t disclosing publicly.

Companies impacted by CrowdStrike outage

To help organizations navigate the CrowdStrike Falcon incident we’ve compiled this list of companies reported to have been impacted by the outage.

^{www.upguard.com}

The US Department of Transport has opened an investigation into Delta over the disruption related to CrowdStrike incident.

Good luck to the CrowdStrike account manager for Delta.

The initial Post Incident Review is out from CrowdStrike. It’s good and really honest.

There’s some wordsmithing (eg channel updates aren’t code - their parameters control code).

The key take away - channel updates are currently deployed globally, instantly. They plan to change this at a later date to operate in waves. This is smart (and what Microsoft do for similar EPP updates).

crowdstrike.com/falcon-content…

By ‘this is smart’ I mean ‘this is smart… now’. Obviously they shouldn’t have been globally, simultaneously deploying kernel driver parameter changes across all customers: it was waiting to go wrong.

They still are btw, as it will take a while to engineer the correct way of doing it.

If you want to know something crazy:

- This year TCS migrated their EDR to CrowdStrike
- Then they announced a strategic partnership with CrowdStrike
- Then they lost all their systems
- They’re just finishing recovery today, 6 days in
- Then they got a $10 Uber Eats voucher
- …which got cancelled due to Uber flagging CrowdStrike’s account as fraudulent

Questions for your EDR providers (do not assume they are experts in availability):

- What are your different update processes?
- How do you test them?
- Do you dogfood test them?
- Do you roll them out in waves? What are the details, eg what percentages and when?
- Do you monitor failures and roll back?

Microsoft are talking about changes to Windows after the CrowdStrike incident. Good.

theverge.com/2024/7/26/2420671…

Microsoft calls for Windows changes and resilience after CrowdStrike outage

Microsoft has started responding with changes it wants to see in the wake of the CrowdStrike botched update. It looks like Windows kernel access is on the agenda.

^{Tom Warren (The Verge)}

There’s a really good discussion on @riskybusiness’s YouTube show about the CrowdStrike incident.

About the 3 minute mark @alex made me realise I was far too kind to CrowdStrike. He rightly rips them apart.

youtu.be/EGRqtscp4eE

Delta are looking to sue CrowdStrike and Microsoft. HT @hrbrmstr

cnbc.com/2024/07/29/delta-hire…

Re the Delta case - the lawyer they’ve hired successfully sued Microsoft previously on behalf of the US government, and the decision was upheld on appeal too. The ruling almost lead to the breaking up of Microsoft.

The following US government backed out of the case.

Bill Gates said at the time the lawyer was “out to destroy Microsoft”.

So there’s a chance here the CrowdStrike incident may end up having implications across vendor industry around warranties etc, we’ll see.