yes! couldn't agree more with your long post. Especially this part: "(This is exacerbated when components of the automation require internal-only tooling—the poor data scientist now needs to go read through a bunch of half-written, out-of-date documentation about tools they simply don't care about to do a task that is not a core responsibility for them.)"

"Vertical integration" in my experience has just been turning a group of simple tools into a complex monolith that no one understands and is extremely difficult to debug

It's also very easy to complain about how employees have résumé-focus in their approach to their work: “why should I bother to learn some internal-only tooling that I'll never use anywhere else (for a task that I don't really even care that much about…)?”

But, to borrow a line from Warren VanderBurgh's ‘Children of the Magenta’: “(in the industry) we created you like this.”

Another key flaw of precomposed automations for rigidly-defined work-flows is that they usually exist in precisely the circumstances that give rise to their own subversion. (I might even go so far as to suggest that the circumstances are the cause of both the mistake and the maladaptive behaviours that address the mistake…)

Ultimately, deep stacks of tightly-integrated components forming a precomposed automation that enacts some work-flow—“vertical integration” as the post frames it—is obvious enough that it seems every big company tries it… only to fail in basically the same ways every time.

Love seeing Children of the Magenta come up!