All technical problems are organizational problems. Put another way: any technical problem is a symptom of an organizational problem.
Even at Google, which has some truly amazing homegrown technical infrastructure, you see what I called Promotion Driven Development ("PDD"). I didn't see this when it came to core technical infrastructure (eg storage, networking) but at a higher level I saw many examples of something being replaced solely (ultimately) because someobody wanted to get promoted and you don't get promoted for maintaining the thing. You get promoted for replacing the thing.
The most egregious example was someone getting promoted to Principal Engineer (T8) for being the TL of something that was meant to replace existing core infrastructure before it had even shipped. In the end it didn't ship. The original thing is still there. But wait, "we learned a lot".
So this happens because the organization rewards the new thing.
So why is your data infrastructure being replaced? Probably because of an organizational failure and it'll have almost nothing to do with technical aspects of that infrastructure. This is true at least 90% of the time (IME).
Data infrastructure is particularly bad for this because in any sufficiently large organization you will completely underestimate the impact of changing data dependencies for metrics, dashboards, monitoring, ML training and so on. Those things can be hard to find and map out and generally you only find them when they break. Sometimes they can break for years before anyone notices even when the thing is used by live production systems.
Even at Google, which has some truly amazing homegrown technical infrastructure, you see what I called Promotion Driven Development ("PDD"). I didn't see this when it came to core technical infrastructure (eg storage, networking) but at a higher level I saw many examples of something being replaced solely (ultimately) because someobody wanted to get promoted and you don't get promoted for maintaining the thing. You get promoted for replacing the thing.
The most egregious example was someone getting promoted to Principal Engineer (T8) for being the TL of something that was meant to replace existing core infrastructure before it had even shipped. In the end it didn't ship. The original thing is still there. But wait, "we learned a lot".
So this happens because the organization rewards the new thing.
So why is your data infrastructure being replaced? Probably because of an organizational failure and it'll have almost nothing to do with technical aspects of that infrastructure. This is true at least 90% of the time (IME).
Data infrastructure is particularly bad for this because in any sufficiently large organization you will completely underestimate the impact of changing data dependencies for metrics, dashboards, monitoring, ML training and so on. Those things can be hard to find and map out and generally you only find them when they break. Sometimes they can break for years before anyone notices even when the thing is used by live production systems.