Internet crash today 19/7/24

[BusyBee]

It's what all these "back doors" in software are about. Getting access so you could do the very thing which someone has inadvertantly done. In effect very similar to the old "Basic" which in this case could be simplified to 10 Print blue screen; 20 GOTO 10. It maybe shows the limitation of the operating system (probably BIOS) in not being able to recognise a possible higher level error and providing a way out but I suppose that would just create further loopholes. I see the issue is that company('s) staff are locked out from being able to interact with the system so are themselves unable to act so IT staff are required for each machine. This is one that hackers all over the world will be viewing closely.

Although full details are not known it is perhaps lucky that military systems are not affected. I have seen it as a worry for some time that so many computers in the world are now connected by single pieces of software. With servers, at the lower end, many users seem to use older software which would give a variable effect (maybe) but Windows itself now demands automatic or prompt action which doesn't leave a lot of scope for user caution. At least with Linux machines you have a stop point in which you can abort any upgrade. I imagine UNIX is the same.

[Nick]

It's absolutely nothing to do with "back doors". It's a 3rd party app that, in some cases, was auto-updating its threat database.

It's also absolutely nothing to do with the OS - the vendor also does macOS and Unix versions, but in this case the version update they screwed up was the Windows one.

There have been plenty of similar episodes involving Unix systems, though more often in the realm of exposing existing or creating new attack surfaces - the histories of OpenSSL and the Apache web server are examples.

It's also nothing whatsoever to do with hackers. It's a legit software vendor getting something basic very wrong. As I mentioned in my previous post, mitigating risk from upstream suppliers should be part of every DR & BC plan. However as also mentioned, it's extremely difficult to get this right every (or indeed any) time. In this case, folk were mitigating the risk from hackers by installing a preventative product (CrowdStrike Falcon) but hadn't mitigated the risk of that product itself being an issue.

Hackers, as the the real ones, not some 14 y/o spotty nerd in their bedroom just copying what someone else had done (aka "script kiddies"), are so far ahead of this trivial discussion that stating 'hackers all over the world will be viewing closely' is frankly laughable.

The press, e.g. the BBC, have largely got this one right. It's not complex. The complex ones never make it to the general press - you have to read the daily security researcher publications and CERT reports to even begin to realise what's going on. Then there's the private invitation-only groups which is where the real discussions happen. The general press are normally weeks behind what's happening on the ground, assuming they report it at all or even vaguely correctly. I know some of these mainstream technology reporters and whilst they are lovely people, when it comes to serious & complex security issues, they are rarely aware or have a clue.

[ppppenguin]

Is there anything special about Windows such that it lacks the ability to recover with some grace from what was a very deep level foul-up? Are Unix-like kernels (in other words Linux, Mac, Android) more robust? Presumably if you prod even the most robust OS in the wrong place it can become FUBAR.

[Mike Watterson]

A canary computer is one that gets all updates, automatically, ASAP. The "production" systems should only get stuff that doesn't kill the canary...
A honeypot is a computer that seems open and ordinary, but actually isn't.

[Mike Watterson]

Is there anything special about Windows such that it lacks the ability to recover with some grace from what was a very deep level foul-up? Are Unix-like kernels (in other words Linux, Mac, Android) more robust? Presumably if you prod even the most robust OS in the wrong place it can become FUBAR.

It's more complicated than that.

In this case "simply"* manually booting in safe mode and deleting the update file makes the computer able to do a regular boot. The fixed file was available about an hour later, so whatever protection the patch was fixing would only be missing till the computer reconnected to Crowdstrike.

* Some windows systems can't either be easily manually booted to safe mode due to being remote in some awkward fashion (remote access only after booting!) or maybe have safe mode disabled for security reasons, or bitlocker configuration.

I was talking to someone doing security at an International Bank. They had actually switched many computers to an alternate product because of performance issues on Windows.

He writes:

Today has been a great day for identifying who uses crowdstrike Falcon

Don't have anything that's a single point of failure.

the reality is that it does not protect against crashes and it isn't a cloud based solution.

If it was more like an Microsoft Defender for Cloud then the impact would have been minimal.

[Nick]

Is there anything special about Windows such that it lacks the ability to recover with some grace from what was a very deep level foul-up? Are Unix-like kernels (in other words Linux, Mac, Android) more robust? Presumably if you prod even the most robust OS in the wrong place it can become FUBAR.

The recovery is actually quite simple - mostly it's just boot into safe mode + networking (though the "networking" bit isn't always necessary), then delete the stuffed file, then reboot.

Problem is that this generally requires someone physically at the machine, and there are many 1,000s affected. Serious servers have a remote console - effectively a remotely accessible console that is embedded in the server hardware and is quite separate from the server s/w. You can use this remote virtual console to manage the hosts just as if your were physically there with a serial console cable and a laptop. HP calls this capability "iLO" (Interactive Lights-out), Dell call it "iDRAC" (Integrated Dell Remote Access Controller), IBM call it "HMC" (Hardware Management Console) etc. It's generally a cost option and not cheap as you have to buy the capability for the hosts plus have a separate "management" network infrastructure to access them, so there's a costs in hardware, licensing and system management overhead, though these costs are mitigated by being able to deal with a large percentage of issues without having to physically go to the host. My teams used to manage 100s of servers this way, many in secure datacentres on the other side of the planet. Without this capability, if we had an issue with a machine in NY, someone would have had to get on a 'plane, get a taxi to Secaucus, get access to the secure underground datacenter there that we used and plug their laptop in. Alternatively you can use "hot hands" which is a local technician in the datacenter who will do what you tell them for a HUGE cost!

There is no defined Unix kernel - Unix can use several kernels, including micro-kernels which are generally safer than monolithic ones as all the drivers etc. are kept apart so one crashing doesn't necessarily impact the system as a whole - unfortunately micro-kernel designs are generally significantly slower than monolithic ones due to the extra overhead of more complex inter-process signalling and context switching, hence are rarely used in "heavy lifting" systems.

The issue here was that in order to catch boot-time hackery, the CrowdStrike Falcon system loads very early in the boot cycle, before any networking etc., and then crashes the system before it can automatically download the fixed patch, i.e. it's in a death loop that has to be broken manually by physical presence at the host or, if available, remotely by a virtual console.

[ppppenguin]

Could the problem be solved, in principle, by running desktops and other end points with a "bare metal" hypervisor which then boots the required OS? This of course assumes that a bare metal hypervisor is relatively immune to attacks, which may be a risky assumption.

Am I right in thinking that even below BIOS level there's a management processor running on all x86 chips? Something that ordinary mortals never see and can't readily access.

[PerdioPal]

I know it wasn't really an internet crash, to those using the affected services it was. I carried a cheque book yesterday just in case!

[Nick]

Could the problem be solved, in principle, by running desktops and other end points with a "bare metal" hypervisor which then boots the required OS? This of course assumes that a bare metal hypervisor is relatively immune to attacks, which may be a risky assumption.

Am I right in thinking that even below BIOS level there's a management processor running on all x86 chips? Something that ordinary mortals never see and can't readily access.

No to the alternative end-point boot. This doesn't get round the TPM issues and is directly contrary to most corporate policies regarding end point security - it opens a whole Pandora's Box of possibilities.

Yes, most modern Intel & AMD processors have embedded management processors (which themselves have been subject to attack).

[Doodlebug]

I have a Dell laptop running W10 and about once a week the
Blue screen death comes up ! Cannot ever read the info on it as it quickly
auto reboots and all is OK again.
I believe that the quick flash of info gives the coding as to what went wrong.

Just a quick note appears that the PC has run into a problem
and needs to restart and it always does OK so I just live with it.