“Zanzibar scales to trillions of access control lists and millions of authorization requests per second to support services used by billions of people. It has maintained 95th-percentile latency of less than 10 milliseconds and availability of greater than 99.999% over 3 years of production use”
"This caching, along with aggressive pooling of read requests, allows Zanzibar to issue only 20 million read RPCs per second to Spanner." ("Only")
I'm surprised by all the numbers they give out: latency, regions, operation counts, even servers. The typical Google paper omits numbers on the Y axis of its most interesting graphs. Or it says "more than a billion", which makes people think "2B", when the actual number might be closer to 10B or even higher.
If your conclusion is "throwing servers at problems" after years of reading the papers about Google infrastructure, you probably are a non infrastructure guy.
A serious conclusion should be that all these infrastructure enable application devs and researchers alike "to throw servers at problem". And these work are exactly the opposite, where they spent years and sometimes even decades meditating the nifty and figure out the most effective and efficient way of utilizing the servers.
It's not servers. It's who will have the best submarine cables. A game in which Apple is not participating btw. Not even with an Elon style moon, err, low earth orbit shot.
It doesn't matter if iCloud is slow or not since most of the interactions with it are in the background. And all of it's content e.g. App Store or Apple Music are cached by CDNs which are hosted in pretty much every country.
And there's much new fun to be had when you have this many servers around. For example - once you start shuffling around tens of petabytes a day you quickly notice that bit flips are very real. Computers do what we tell them to do with incredibly high probability, but it is always below 1.
Impressive!