> all software goes down and requires specialist intervention eventually
Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional?
If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right.
If there are 100 critical systems running at 3 9’s, you’d expect 2.5 hours of downtime per day.
So yeah, all software should keep running. But it doesn’t. And something like Twitter isn’t “a software”, it’s a very large assembly of lots of different software systems and the exponential math that dependencies create.
Yep, and when one of the SEVs rolls around that would have been small (say 5m of downtime fixed with a flag flip), it instead will have a nontrivial chance of escalating into a major multi-hour/multi-day outage without the right institutional knowledge.
I'd guesstimate that Twitter probably has dozens of services that are in the critical path of an average user interaction. It's hard to keep even logically optional dependencies truly optional in large scale systems involving many people.
However Twitter didn't die in the past when fail whales ruled its day, so they probably won't kill it now. It's just not that kind of business. (In contrast, a one hour outage had me directly apologizing to our largest customers on the phone). That said, Twitter can only be unstable and lack feature growth for so long before something else takes its place, so Musk is on a clock.
Right, but Twitter wasn't a healthy business (in the sense of being profitable most years) in the first place so it's not beyond the realm of possibility they took reliability further than made sense. Anyway they now have a huge debt load that changes the calculus regardless.
Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional?
If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right.
If there are 100 critical systems running at 3 9’s, you’d expect 2.5 hours of downtime per day.
So yeah, all software should keep running. But it doesn’t. And something like Twitter isn’t “a software”, it’s a very large assembly of lots of different software systems and the exponential math that dependencies create.