Well, I find this post looks good, but a like lot of 'data for developers' posts it's just a list of tools. As if a collection of tools banded together actually makes your customer successful.
What's missing?
1. There's nothing about deployment. How do I take this collection of tools and code and actually deploy it into production, or actually regression test it functionally? How do I make a small change in a database table and not have a massive regression? How do you do that automatically? How do you do it quickly?
2. It's cursory on testing. One of the biggest differences from a software developer to a data engineer is that your data providers give you crap data all the time. It could break. How do you test data? How do you get adequate test coverage? These things are essential for software developers and are actually doubly essential for data engineers and building analytics systems.
3. It's what success looks like. It's not just about a collection of tech; it's about making your customers successful. What does it mean to deliver good insight? How do you do it? How do you measure customer success, and measure your success? As a team, you wouldn't talk about software engineering without mentioning DevOps or DORA metrics. There's nothing here about that.
The dude has obviously never met a multimillionaire owner of an auto body shop, a house demolition business, or a plumber. Did they do well in high school calc class? No. Are they richer than the author? Yes
Blowing up the cognitive hierarchy is a gift that AI gives us. Let's move into an age where hard work and character matter more than your SAT score at 17.
I think cognitive ability is a real thing with a genuine genetic component; twin studies make that pretty hard to deny. But the SAT measures a narrow slice of ability under conditions that favor prep access, and speaking from experience, scoring well on standardized tests without having to work for it hurt me long term. I coasted on test-taking ability and didn't develop real discipline until adulthood.
Without standardized tests, my transcript probably doesn't get me into a top school, I likely don't end up in tech, and honestly maybe I would've been forced to develop a work ethic earlier, which might have been better for me.
> Why do you think a high SAT score doesn't need "hard work and character"?
Well, it absolutely doesn't need both, because I got one without, at least, the first of those (beyond the extent that "getting up early on a Saturday" and "sitting calmly while bored out of my mind after finishing each portion of the test waiting for time to expire" is "hard work".) I like to think I had the second, but it didn't seem particularly relevant to the test in any way.
The SAT, like general intelligence, is half hard work and half inherited ability, from what I've read. Characteristics that you have no control over, like height, intelligence, race, or sex, should not determine your wealth. It should really be about hard work and character; that's fair. At least in the U.S., there are too many industries like law, venture capital, or VC-invested startup founders, where the pedigree of your school is what matters to your success.
AI automating (almost) all white collar work will concentrate all those salaries into the hands of a very small group of people.
BTW the number of plumbers who become multimillionaires is vanishingly small, while the number of SWE who have in the last 15 years is enormous by comparison.
Neither statement is true, unfortunately. The reports that you read in the media don't reflect the state of the world. In the US, most plumbers and SWE make between 100-200K. The way to wealth is to start one's own business. Hence, hard work and character.
Many data teams often find themselves as 'tool jockeys' instead of becoming true engineers. They primarily learn some company data, and then rely on drag-and-drop or YML configuration functionality within the constraints of the tool's environment.
Their organization often insists they must use standard tools, and their idea of a good job is that the task works fine within their personal version. No automatic testing, no automated deployment, no version control, and handcrafted environments. And then they get yelled at when things break and yelled at for taking too long. And most DEs want to quit the field after a few years.
The real question is not that DE and software engineering are converging. It's why most DEs don't have the self-respect and confidence to engineer systems so that their lives don't suck.
Prefacing this with an acknowledgement that I'm a public sector data analyst by trade so my experience may not be universal.
My view is that it isn't so much a lack of "self-respect and confidence" but an acknowledgment that the path of least resistance is often the best one. Often data teams are something that was tacked on as an afterthought and the organizational environment is oriented towards buying off-the-shelf solutions rather than developing things in house.
Saying that, versional control and replicable environments are becoming standard in the profession and, as data professionals become first class citizens in organizations, we may find that orgs orient themselves towards a more production focused environment.
“Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space.” -Douglas Adams
I really appreciate how functional the building is. It's extremely visually distinct while having really engaging vertical elements (I've always thought it evoked waterfalls) and lacking the functional flaws I've seen with other highly visible architecture (I'm especially thinking of the Gehry Building at MIT - that's whimsical in appearance but an absolute nightmare of usability with awkward unusable interior spaces and a long legacy of mold and maintenance issues).
It's especially amusing that Boston City Hall is within a stone's throw of the only block that survived the fire of 1872 and throws a shadow over Faneuil Hall.
Stata at MIT has sort of grown on me from an abstract architectural perspective. But it cost a lot and I've never really heard good things about it from people who actually use the building though I've never been in it myself aside from the ground floor. It was also sort of justified as a landmark northeast entrance to campus but was soon pretty much literally overshadowed by a lot of newer construction in the area.
sigh