Academia's the biggest fucking scam there is. Look at the prize. Around age 35-40, if you work 80-hour weeks and don't make any mistakes, the prize is getting a job that you can't be fired from. Meanwhile, the same kind of exertion (and political luck) in finance or software will have you able to retire: not in luxury, but at a middle-class standard of living that could be described as "making your own tenure". Also, you get to live where you want, instead of rolling the dice on where in the country you will end up living.
It's billed as "the life of the mind", but the reality is that a lot of the work is mindnumbingly boring: writing grant applications, grading papers, attending committees. Every industry has some boring work and vicious politics, but most industries are honest about it. In software, having been fired once because of political bullshit is par for the course. Yep, that happens. In academia, getting shot down for tenure is this huge mark of shame.
Finally, the academic industry has a huge underclass of people who spend lots of time between adjunct gigs on teaching work that is viewed as a commodity and paid extremely poorly.
I'm sorry, but this industry has sold out a generation and a half, while tuitions have shot toward the moon, and that's fucking inexcusable. University leadership: stop building fucking ziggurats and colossuses and get fund some goddamn research and teaching. In other words, do your job.
Meanwhile, the same kind of exertion (and political luck) in finance or software will have you able to retire: not in luxury, but at a middle-class standard of living that could be described as "making your own tenure". Also, you get to live where you want, instead of rolling the dice on where in the country you will end up living.
In this realm, I really don't think software, finance and academia are all that different. You go where the work is, and on average, the best you can do is to nicely secure a comfortable, upper-middle class position for yourself with plenty of job security.
You're right that being lucky enough to get a tenured position, but "applying that luck" to software or finance, can make you a millionaire. Problem is, that kind of luck is not a known probability on which we can set rational expectations. If I go work 80 hours/week at a software start-up, I can't expect even a 3% chance of becoming a millionaire. It's just not a random process with a set probability distribution. So you can't "apply your luck" to software instead of academia, because we're talking about two different kinds of luck.
This is a good point. The types of people who thrive in one environment are not necessarily going to do well in another.
My issue with academia is that it has minimal job risk but high career risk. So, while it's rare that someone actually gets fired (I've seen 9th-year grad students lose funding; I guess that counts) the risk of ending up in a ruined career is high. On the other hand, in software and finance getting fired is practically a rite of passage-- happens to everyone, especially ambitious overperformers-- and it's painful but it doesn't end your career.
On the other hand, one thing academia has going for it is career coherency. In industry, you serve a boss and have to make it look like you value his political aspirations over your own career development. In academia, your job is to invest in yourself-- to publish and develop a reputation, to attend conferences, to improve your skills. So that's one thing that's nice about it.
Ok, so here's the question: what options could I have in software, where I can actually have a career and get comfortable and eventually even raise some kids, that don't involve me in awful levels of politics/deception (particularly regarding my own "career coherency", as you put it), and don't bore me to death?
Because I've programmed for a hobby since age 11, and I've programmed a few times "for reals" in actual paid work, and while I've every bit of respect for the craft of programming, merely doing that every day under industrial production conditions drives me a little out of my mind.
I won't finish my research-track MSc for a while, so I don't have to definitely choose industry or academia for a while. Still, I'm back doing this because I found that without a higher degree or industrial experience in such, you really can't get into the cooler, higher-level jobs in "the real world" either. If you get out of school and code for a living, you will eventually find yourself locked out of the deep wizardry of computing (systems programming, programming languages, networking protocols, security, etc.)... unless you find just the right company willing to take a chance on you.
Yes, getting the Master's degree is a good call, for the reasons you mentioned. PhD bigotry is pretty severe in a lot of places.
CS academia is less scammy than other disciplines, because of the high-quality exit options. You're not doing wrong by going for the MSc. That's a good call.
Regarding being "locked out", I think the new rules are:
* network aggressively at all times. When you join a big company, network internally so you can get a decent transfer after 6+ months. You won't get hired on to the ML project at the front door. Nor will you get it through official channels. Network aggressively and find someone who will request you.
* keep learning. Eventually, you'll stand out as the guy who's 30+ and keeping abreast of cutting-edge software trends, and that has its own kind of impressiveness.
It's harder than it should be to get interesting work, because the world is run by idiots, so there isn't much tolerance for interesting stuff. I'd love to see that change, but for the mean time, you just have to figure out how to play the world that is. Stealing an education from work is usually a good idea (don't consider it "deception"; honest people mouth off and get fired.) Keep Learning and Carry On.
Well, bizarrely, my internship hunt for this summer is actually going pretty well. Got about 4 companies I'm interviewing with at this point, one of which is The One that I absolutely want to work for. Not as a Final Career Destination or anything, but I think they do some of the best and most interesting work on Earth, so I'm very interesting in seeing how they've built a business out of the stuff.
CS academia is less scammy than other disciplines, because of the high-quality exit options. You're not doing wrong by going for the MSc. That's a good call.
Actually, in this case I think it's a good idea because even the coursework here seems very devoted to building artifacts. This semester's project in my Coursework Course is building a static analyzer for LLVM IR of C code. Next semester I'm going to try to take Advanced Operating Systems, in which project groups build a small OS from the ground up.
These aren't exactly start-ups, but they're the kind of coursework that generates project code which you can throw on Bitbucket and use as proof not only that You Can Code but that You Can Do Advanced/High-Level Work.
Surprisingly, my undergrad institution wasn't very good at that.
It's billed as "the life of the mind", but the reality is that a lot of the work is mindnumbingly boring: writing grant applications, grading papers, attending committees. Every industry has some boring work and vicious politics, but most industries are honest about it. In software, having been fired once because of political bullshit is par for the course. Yep, that happens. In academia, getting shot down for tenure is this huge mark of shame.
Finally, the academic industry has a huge underclass of people who spend lots of time between adjunct gigs on teaching work that is viewed as a commodity and paid extremely poorly.
I'm sorry, but this industry has sold out a generation and a half, while tuitions have shot toward the moon, and that's fucking inexcusable. University leadership: stop building fucking ziggurats and colossuses and get fund some goddamn research and teaching. In other words, do your job.