Internet crash today 19/7/24

[Mike Watterson]

I know it wasn't really an internet crash, to those using the affected services it was.  I carried a cheque book yesterday just in case!

No-one here in Ireland, anywhere, shops, services or State agencies, will take cheques, since years ago. There was a fuss for a while because the NCT (=MOT) wouldn't take cash. Most banks won't take them, especially Sterling. IBAN is the only alternate to paypal, cards or cash. Many (most? all?) banks don't even issue cheque books. The stamp duty per cheque may have helped to kill it.

Also amazingly the old Sort code still exist, but in UK and Ireland the account number can exist at a different bank and the Sort code has no check digit. So never use Sort + local account code, only IBAN, which is mostly free (not for non-EU state transfers) and cheaper than Paypal or Credit card if there is an IBAN charge (if not using express).

[Nick]

IBAN is not a service, it's your International Bank Account Number and if you look closely, in the UK at least, it contains your bank name , sort code and account number. In Eire, the format of your IBAN is very similar. See https://www.iban.com/

In the UK, examples of services that use IBANs as account identifiers to send and receive money are: BACS (Bankers Automated Clearing Service - mainly used for inter-bank transfers for Direct Debit/Credit) and FPS (Faster Payment Service - mainly used by individuals to transfer funds).

[Mike Watterson]

Nit picking.

People say "Pay by IBAN". It's the code the service uses and it's less error prone than using sort and account code, where often people have paid to the wrong bank.

Using it has replaced cheques here. Popular also in German y to avoid paypal fees on ebay. Also works to pay credit card. No cash in hand now to builders etc. IBAN.

And if we want nit pick, it's Éire and only when using Irish. The English name of the whole island and also the Nation state is simply "Ireland" (Republic of Ireland is a nickname).·

[Mike Watterson]

Interesting that there is a lobby group to keep cash
https://www.theguardian.com/technology/a...e-alliance

However there needs to be diversity and resilience. The actual POS can't take cash if their barcode scanning, scales, tills and Just-in-time ordering isn't working.

[BusyBee]

It's absolutely nothing to do with "back doors". It's a 3rd party app that, in some cases, was auto-updating its threat database.

It's also absolutely nothing to do with the OS - the vendor also does macOS and Unix versions, but in this case the version update they screwed up was the Windows one.

There have been plenty of similar episodes involving Unix systems, though more often in the realm of exposing existing or creating new attack surfaces - the histories of OpenSSL and the Apache web server are examples.

It's also nothing whatsoever to do with hackers. It's a legit software vendor getting something basic very wrong. As I mentioned in my previous post, mitigating risk from upstream suppliers should be part of every DR & BC plan. However as also mentioned, it's extremely difficult to get this right every (or indeed any) time. In this case, folk were mitigating the risk from hackers by installing a preventative product (CrowdStrike Falcon) but hadn't mitigated the risk of that product itself being an issue.

Hackers, as the the real ones, not some 14 y/o spotty nerd in their bedroom just copying what someone else had done (aka "script kiddies"), are so far ahead of this trivial discussion that stating 'hackers all over the world will be viewing closely' is frankly laughable.

The press, e.g. the BBC, have largely got this one right. It's not complex. The complex ones never make it to the general press - you have to read the daily security researcher publications and CERT reports to even begin to realise what's going on. Then there's the private invitation-only groups which is where the real discussions happen. The general press are normally weeks behind what's happening on the ground, assuming they report it at all or even vaguely correctly. I know some of these mainstream technology reporters and whilst they are lovely people, when it comes to serious & complex security issues, they are rarely aware or have a clue.

I think you totally misunderstand my take on this. I was actually comparing the result of simple earlier hacks with the result of this failure, implying that even with complex protection the end result was somewhat similar. My referral to other operating systems was to wonder whether their lesser deference to outside control would improve their protection or recovery options.

I can see what happened and wonder how my old work collegues would be coping now. It is perhaps lucky that safe mode is still accessible on the PC's. I remember one time on my work PC when it got a denial of service bug where it wasn't and it took split second reactions to take out the malware before it booted up. Still that's another time.

Not being directly involved with an IT department I will stand aside now as my take is perhaps a bit out of line but I just think a bit in parells and possibilities.

Tracy

[Radio Fixer]

But why did CrowdStrike put the 'fix' out World wide? Was it just arrogance not to try it small scale locally as most cautious folks would do?

Gary

[Nick]

But why did CrowdStrike put the 'fix' out World wide?  Was it just arrogance not to try it small scale locally as most cautious folks would do?

The actual fix is trivial and was issued within about an hour, so those systems that were still connected and weren't crashing fixed themselves. The problem was that some folk a) didn't have access as their systems kept crashing, and b) the trade press was on to them about what the fix was. It's not a big deal.

[Radio Fixer]

Sorry, my text wasnt clear.

please replace fix with upgrade

Why put this out world wide without trying it small scale first ?

Gary

[Nick]

Sorry, my text wasnt clear.

please replace fix with upgrade

Why put this out world wide without trying it small scale first ?

It would have been tested prior to being made public. Screwups like this one mostly happen when humans are involved in the process. The Facebook BGP router configuration debacle is another example - there are many others, some not widely known.

Most modern development cycles will use a continuous integration (CI) methodology where every time any sort of change is made to a development environment (code change, configuration file change etc.) a build is automatically started and if that succeeds then a QA/regression test suite is run and if THAT passes, the build may be accepted for alpha testing with a tame group of users, then beta testing and finally general release. Whenever a new failure mode is identified, a check for that will be added to the test suite. What you're aiming for here is "coverage" - the automated and other tests should test EVERYTHING, i.e. 100% coverage, however this is an unreachable ideal - all you can do is try to get as close as you can to 100% and treat the test suite as a live project in itself. Many organisations will have a completely separate QA team to create the test suites as developers are often considered "too close" to the code and may be subject to unconscious bias (assumptions) when creating tests. If you fail at any point, the cycle restarts.

It may be that this issue snuck through as it exposed a flaw not seen before and the test suite didn't catch it. Over the years teams I've run have mostly used the Atlassian tool suite to manage most of this, particularly Jira, Bamboo, Crucible & Confluence plus other tools e.g. SmartBear TestComplete which integrate with the Atlassian tools.

Many vendors with large user bases have release "channels". The "stable" channel will be what 99.9% of all users and production systems are on and they will only get updates when everybody else upstream (CI, alpha, beta etc. see below) is happy. There are sometimes "feature" channels which if you are on will allow you to get early access (at your own risk) to new functionality, then there are "alpha" and "beta" channels to which you generally have to apply to the vendor in order to be accepted (they will only want tame & qualified folk with the right infrastructure on those channels and they will have to sign NDAs etc.). Note that there are no set names for these channels and each vendor might use a different name, but you get the idea.

This whole process is called "release management" and is where CrowdStrike failed so spectacularly.

Many end-user organisations will have all their product infrastructure on the "stable" channel but will have test environments which might be on "feature", "alpha" or "beta" etc. or even on "stable" to do their own testing before "stable" is rolled out to the production systems. The more testing you do as a end-user, the more people & infrastructure you need and the more expensive your IT costs. I've been lucky in working in the financial world in various countries - money for this stuff is generally not an issue as the consequences of downtime is generally more costly. There is a law of diminishing returns though, so there have to be constraints on testing - time, money, resources, business pressures etc.

There's a fundamental tenet for most organisations: "If it ain't broke, don't fix it", so many software updates may be skipped if the release notes don't indicate something in the release that is of benefit to the organisation, e.g. a fix to a bug they've seen, or introducing a new feature that is needed.

The problem with cybersecurity is that the environment changes very very quickly - hourly or faster - so to keep on top of new threats, updates to production environments have to be issued VERY frequently, often daily or even intra-day. This makes pre-stable-release testing an interesting challenge as the time window to get stuff 100% right is small and with that sort of pressure, mistakes can happen.

EDIT: What I suspect happened was that a fairly tame change to a configuration file (which is all that was in the update that caused the issue) exposed a coding flaw in the kernel-level driver that CrowdStrike use and which is loaded early in the Windows boot process (in order to catch nasty viruses etc. - generally the earlier you can load, the better)
The flaw itself may have been there a long time and is probably something like a buffer overflow causing a stack corruption, or memory pointer being corrupted etc. I admit this is supposition, but a configuration file change which introduced a longer-than-previously-used string or similar and which in itself may seem harmless, could easily expose this type of flaw. It's a coding error, a bug, which was not caught in testing and which was not or could not be caught and handled safely by the driver in question, hence the BSOD and reboot.

[ppppenguin]

So potentially an old fashioned buffer overflow or dangly pointer sort of problem.