Everytime i look into such catastrophic issues, it always boils down to lack of robust canarying mechanisms.

They have enough client base that they can even run an A/B test on the whole binary level, but no.