"software supplier"??? Why on God's green earth isn't someone familiar with the code on 7/24 pager duty for a system with this level of mission criticality?

That would be... the software supplier. This is quite a specific fault (albeit one that shouldn't have happened if better programming practices had been used), so I don't think anyone but the software's original developers would know what to do. This system is not safety-critical, luckily.

I think there is a bit of ignorance about how software is sold in some cases. This is not just some windows or browser application that was sold but it also contained the staff training with a help to procure hardware to run that software and maybe even more. Such systems get closed off from the outside without a way to send telemetry to the public internet (I've seen this before, it is bizarre and hard to deal with). The contract would have some clauses that deal with such situations where you will always have someone on call as the last line of defense if a critical issue happens. Otherwise, the trained teams should have been able to deal with it but could not.

My jaw kept dropping with each new bullet point.

Same, is aviation technology really this primitive?

It is mostly quite primitive, but it also works amazingly well. For example ILS or VOR or ATC audio comms can all be received and read correctly using hardware built from entry level ham radio knowledge. Altimeters still require a manual input of pressure. Fuel levels can be checked with sticks.

Kinda the opposite of a modern web/mobile app, complicated, massively bloated and breaks rather often :).

It's worse than you know. Ancient computer systems, non-ASCII character encodings, analog phone lines, and ticker-tape weather.

You'll also be surprised to learn there's still parts of the US where there's no radar or radio coverage with ATC, if flying at lower altitudes. (Heck, there's still a part of the Pacific Ocean that doesn't have ATC service at any altitude.)

Aviation drove a lot of the early developments in networked computing, which also means there's some really old tech in the stack. The globally decentralized nature of it all and it being a life-critical system means it's expensive and complicated to upgrade. (And to be clear, it does get upgraded - but it in a backwards compatible way.) Today's ATC systems need to work with planes built in the 1950s, and talk to ATC units in small countries that still use ancient teletype systems and fax machines.

But yet it's all still incredibly safe, because the technology is there to augment human processes - not replace them. Even if all the technology fails, everything can still be done manually using pen and paper.

You might find it interesting that the SF subway runs on floppy disks. Not the fancy new 3.5" ones, either.

https://sfstandard.com/2023/02/02/sfs-market-street-subway-r...

And what non-primitive software do we have that is reliable? None that I know of.

Airline messaging is wild, this blog from 2010 knows what’s what: https://cos.livejournal.com/79455.html

shhh, nobody tell xvector that unleaded avgas finally happened in 2022 :)

Thanks for the summary and TL;DR.

Essentially this is down to the lack of proper namespace, who'd have thought aerospace engineer need to study operating systems! I've a friend who's a retired air force pilot and graduated from Cranfield University, UK foremost post graduate institution for aerospace engineering with their own airport for teaching and research [1]. According to him he did study OS in Cranfield, and now I finally understand why.

Apparently based on the other comments, the standard for namespace is already available but currently it's not being used by the NATS/ATC, hopefully they've learnt their lessons and start using it for goodness sake. The top comment mentioned about the geofencing bug, but if NATS/ATC is using proper namespace, geofencing probably not necessary in the first place.

[1] Cranfield University:

https://en.wikipedia.org/wiki/Cranfield_University

It sounds like a great place to study that has its own ~2km long airstrip! It would be nice if they had a spare Trident or Hercules just lying around for student baggage transport :)