It's a very long post with a mix of technical (math) and philosophical sections. Here are the most striking points to reflect upon IMHO.
> It seems to me that training beginning PhD students to do research [...] has just got harder, since one obvious way to help somebody get started is to give them a problem that looks as though it might be a relatively gentle one. If LLMs are at the point where they can solve “gentle problems”, then that is no longer an option. The lower bound for contributing to mathematics will now be to prove something that LLMs can’t prove, rather than simply to prove something that nobody has proved up to now and that at least somebody finds interesting.
Training must start from the basics though. Of course everybody's training in math starts with summing small integers, which calculators have been doing without any mistake since a long time.
The point is perhaps confirmed by another comment further down in the post
> by solving hard problems you get an insight into the problem-solving process itself, at least in your area of expertise, in a way that you simply don’t if all you do is read other people’s solutions. One consequence of this is that people who have themselves solved difficult problems are likely to be significantly better at using solving problems with the help of AI, just as very good coders are better at vibe coding than not such good coders
People pay coders to build stuff that they will use to make money and I can happily use an AI to deliver faster and keep being hired. I'm not sure if there is a similar point with math. Again from the post
> suppose that a mathematician solved a major problem by having a long exchange with an LLM in which the mathematician played a useful guiding role but the LLM did all the technical work and had the main ideas. Would we regard that as a major achievement of the mathematician? I don’t think we would.
> by solving hard problems you get an insight into the problem-solving process itself, at least in your area of expertise, in a way that you simply don’t if all you do is read other people’s solutions. One consequence of this is that people who have themselves solved difficult problems are likely to be significantly better at using solving problems with the help of AI, just as very good coders are better at vibe coding than not such good coders
Yes but it's not just that if you solved a problem yourself, you're better at solving other problems; it's also that you actually understand the problem that you solved, much better than if you simply read a proof made by somebody (or something) else.
I see this happening in the enterprise. People delegate work to some LLM; work isn't always bad, sometimes it's even acceptable. But it's not their work, and as a result, the author doesn't know or understand it better than anyone else! They don't own it, they can't explain it. They literally have no value whatsoever; they're a passthrough; they're invisible.
Are you a cutting edge research scientist or something? Everyone I know works in the same domain every day. The problems are the same. People aren't solving brand new problems to humanity every day. We make budgets and look at ticket counts. Roll out patches. Replace hardware. Upgrade software packages. Make a new dashboard to track a project. I guess if every day is a completely novel thing for you, ok. I feel like the goalposts have moved to an absolutely ridiculous place. Oh no, I won't have a bunch of random error log numbers memorized anymore? Who gives a shit. I just want to afford a place to live so I can play my guitar and make something good for dinner. Maybe I'm just old, but I don't see why the average person needs to be a fuckin genius problem solver.
I think that’s fine, but 1) that mentality leaves you extremely vulnerable to being disrupted by LLMs and 2) IMO, if you are solving the same problems every day it means you are not making progress on solving the root causes of those problems. What you are describing is toil, not knowledge work
I don't think it matters much what kind of problem it is. If it is challenging enough to benefit from assistance and you end up playing a minor role in the solution, it seems like you are putting yourself in the worst position possible. You lose your edge for functioning within the problem space and it raises the question why you are even in the loop at all. If its job security you want, transforming your role into LLM babysitter seems like the worst way to ensure it.
It's an adversarial economy. Using a LLM at work doesn't mean the work is challenging. A lot of jobs are "bullshit jobs". People are using LLMs because it gives them back time. If they don't use it their colleague will and make them look bad.
Company might fire you tomorrow. Fundamentally if a LLM can do the job it's not just employees at risk, it is also the company. There is a lot of symmetry actually with how companies delegate to employees to how employees delegate to LLMs. You can follow the logic to conclude a lot of companies are then bullshit companies. This is not a problem for the individual to solve. Your job at work is akin to the company's - earn the best return while you still can. Wasting your time for the essentially the same output at a slower pace is a bad return.
When people get laid off en masse this incentive structure will have to be altered. But telling an individual to ignore their basic economic incentives until then is unlikely to work.
I have also come to the conclusion independently that a lot of companies are bullshit companies, maybe that is closer to the core issue. For the individuals who do have some choice in the matter, I think it is important to hold on to their skills by continuing to use them. It sucks that our work culture is so competitive, but from that angle I believe they will stand out eventually as more competent.
Most companies are real, it's just that a good fraction of the work is mostly unnecessary. Partially because of the overhead of doing business activities that is unneeded most of the time, partly because we don't know what work will be useful, and partly for silly social reasons
I keep coming back to the idea that all the upheaval combined with all the new tools at our disposal will empower and motivate people to start businesses that challenge the status quo. I've lived long enough to see that play out at scale, it is basically how we got Google. That might not sound encouraging, but Google was once a really inspiring company and one of the best places to work.
Ok let's make math illegal and burn down the data centers I guess. Idk what to tell you, but we will adapt and new roles will be created. Just like every single tool and piece of tech that came before. LLM manager? Fine.
The parent said American corporations. No one with any sense wants a dependency critical to their state or private company sitting under the direct control of America any more.
I think it's intellectually dishonest to dismiss the absolute accumulation of human's knowledge under very specific brands for profitability using false equivalencies. When I build something using chatGPT, especially if I was unable to build it before, I arrive at a result that I could have previously arrived with "hard work" by skipping the "hard work" part.
Now, many will argue that you wouldn't have poured in time and energy in that endeavour anyways, so it's fine. But the crucial part missing here is the effort. We're about to witness the side effects of societal-wide reliance on LLM's, the same way we're still paying the price for the social media boom, misinformation, propaganda, echo-chambers and algorithmic bubbles.
Notice that none of the above actually invented misinformation, etc. they just magnified an existing problem. LLM's magnify the need to "get it done, fast" but I don't see the engineering excellence everyone promised me that I'll see at any level.
In the US, much of the woods are owned by corporations too. Those that aren't are, in theory, owned by the public, but the oligarchs work hard to hollow that out so that practically public lands are owned by them too.
>Just like every single tool and piece of tech that came before.
The thing about relying on the past to predict the future is that works ... until it doesn't.
We've yet to see a technology with as diverse utility as LLMs. What happens when not just the tech sector starts downsizing, but the whole white collar workforce?
In the past, one such "new role" was that of slave. In fact, we expect slavery is <10,000 years old! Yes, new roles will be created. But there's nothing to say that they'll be pleasant for us to take on.
It doesn't seem like you're responding to my post, more to the quote? But my point isn't that everybody should be a genius problem solver, although that would help, while being stuck in the same routine doesn't.
My point is, if you delegate your job to AI, and it works, then 1/ you don't know the result of the work in more detail than any other person, and 2/ the people you're reporting to can probably write a prompt as good as yours, if not better.
Which means: you've made yourself dispensable. Nothing very good for dinner; no nice place to live. But lots of time to practice guitar I guess.
>Who gives a shit. I just want to afford a place to live so I can play my guitar and make something good for dinner. Maybe I'm just old, but I don't see why the average person needs to be a fuckin genius problem solver.
I enjoy programming and want to be engaged for the 40 hours a week where I sell my labor.
I also care about my profession and technology, and I don't want the world to become an idiocracy where nobody understands any of the technology we're overly depedent upon.
> I see this happening in the enterprise. People delegate work to some LLM; work isn't always bad, sometimes it's even acceptable. But it's not their work, and as a result, the author doesn't know or understand it better than anyone else! They don't own it, they can't explain it. They literally have no value whatsoever; they're a passthrough; they're invisible.
According to the blog post linked in the OP, the LLM-generated results were read, understood, and confirmed by the mathematician whose work they built on.
I notice a dichotomy here between people who care about results and people who care about process. The former group wants to use LLMs insofar as they can contribute to getting results. The latter group is wary of LLMs because they're more interested in the process and less interested in the results themselves. Needless to say, I think the former group is right, and I'm happy to see that mathematicians (or some of them) agree.
I think you are misunderstanding the parent's comment.
>the LLM-generated results were read, understood, and confirmed by the mathematician whose work they built on.
The mathematician and the blog author are not the same person (as you seem to understand). Nathanson (the mathematician) is the one who is the expert verifier. He is the person who has the higher value and won't be fired in some hypothetical.
>>They don't own it, they can't explain it. They literally have no value whatsoever; they're a passthrough; they're invisible.
This is the blog author in the parent's description. If their boss asks them what they need to prove that the AI is more than capable in this domain and the author tells their boss they need Nathanson (the mathematician) to verify the results, his boss will thank him for demonstrating the AI's capability in this domain, fire him, pass his prompt history to Nathanson, and keep Nathanson on the job (the expert verifier).
Which is the parent's point after all, because he's referring to the hypothetical job security of the blog author not the mathematician.
> The mathematician and the blog author are not the same person
> (as you seem to understand). Nathanson (the mathematician) is
> the one who is the expert verifier. He is the person who has
> the higher value and won't be fired in some hypothetical.
The creation of the system is deeply impressive, so are compilers but I don't raise a toast to it each time I build my code. Like generated art, people aren't going appreciate it on the same level.
To each their own. I mean compilers didn’t produce trillions of dollars of investment, and produce serious and profound philosophical questions about the nature of consciousness but you’re right, thank god we have C
Compilers just made it all possible, but they are not new and shiny. LLMs did not produce the philosophical questions, but they do raise them. It's worth noting that computers have been changing the way we think about consciousness long before LLMs, largely thanks to compilers.
I don't think the level of investment in an idea is equivalent to how impressive it may be. Most of the investment in AI is based on the idea that it will make professions and human labour obsolete, which means whoever has the reins at the moment it "solves" the "problem" of human labour will effectively reign over everyone else. The level of investment is then somewhat orthogonal to how technically impressive it is.
Not to mention that the less easily-explainable a technical achievement is, the less investment it will attract simply because fewer people will grasp the ramifications. You can describe AI in two words ("machine human") while it would take a few more to describe compilers in an instantly understandable way.
I mean - I'd say electricity, agriculture, steam power, metallurgy, silicon computing (cmos), atomic power, the scientific method - these are _all_ very impressive - all lead to drastic changes for humanity. Not sure how I'd rank them.
I personally think AI will end up sitting in the top 3 of these - but that is an opinion. I do think it is obvious it is at least _somewhere_ in that list.
What a weird fucking measuring stick. By your logic crypto is one of man's greatest achievements because it received oodles of investor cash, and kicked up tons of conversations online about the nature of finance and banking.
It’s not the measuring stick, what it can do is the measuring stick. Another person comparing a system that can do Erdos proofs to a compiler or even worse, crypto? Everyone has a right to be unimpressed i just find it incredible to be so dismissive with a straight face.
Sure, but the point is that at some point (e.g. when starting a PhD) one needs to do research, not learn the basics. And LLMs make that harder, because they solve the "easy research" part.
Take a young lion "fighting/playing" with another young lion as a way to learn how to fight, and later hunt. And suddenly they get TikTok and are not interested in playing anymore. Their first encounter with hunting will be a lot harder, won't it?
> People pay coders to build stuff that they will use to make money and I can happily use an AI to deliver faster and keep being hired.
Again, that's true but missing the point: if you never get to be a "good coder", you will always be a "bad vibe coder". Maybe you can make money out of it, but the point was about becoming good.
1. It matters because there are human mathematicians who pride themselves for their mathematical achievements. Mathematics is art to them.
2. Yes, it is. Because pre-LLM era computer-aided proofs were about using the computer to either solve a large number of cases or to check that each step in a proof mechanically follows from the axioms.
It matters because most mathematicians thrive on the recognition of their achievements. If what you do any mediocre mathematician could have done, that takes away motivation and fulfillment.
> Training must start from the basics though. Of course everybody's training in math starts with summing small integers, which calculators have been doing without any mistake since a long time.
Yeah, it's the same way with learning programming. LLMs can handle basic programming (and increasingly advanced programming) but I think it's necessary to write code by hand. As a beginner, of course, and arguably to maintain skill later too.
The alternative would be like, just asking ChatGPT to do your math homework and then "verifying" it by looking at it and saying "yeah, that looks okay." What are you going to learn?
You only get good at the things you actually do. Our ancestors had to maintain a minimum level of fitness in order to be able to eat -- a level that most people today never reach, because the modern world has removed that need. Thinking is a skill just like any other, so what happens when people no longer have to exercise that skill to survive? It's a scary thought.
Tools don’t eliminate work, they abstract and amplify work. Those who miss this point are doomed to become the folks who say “back in my day we walked to school in the snow, uphill both ways”.
The map isn’t the territory; thinking about what to build is just as valid as thinking about how to build it. Architects aren’t carpenters, but that doesn’t mean there’s no value in architecture.
Not following. The demand for architects is gated by the cost of building. And the metaphor is that all if us who used to be carpenters can be architects, in the software sense. Maybe some people don’t want to be, but it is still a very thought-intensive profession.
The question is whether vibe coding requires a lot of thought, and I don't believe it does. The industry in in full blown idiocracy at the moment, and if you think you're a real engineer despite not understanding what you're building, you're a joke.
> It seems to me that training beginning PhD students to do research [...] has just got harder, since one obvious way to help somebody get started is to give them a problem that looks as though it might be a relatively gentle one. If LLMs are at the point where they can solve “gentle problems”, then that is no longer an option. The lower bound for contributing to mathematics will now be to prove something that LLMs can’t prove, rather than simply to prove something that nobody has proved up to now and that at least somebody finds interesting.
Training must start from the basics though. Of course everybody's training in math starts with summing small integers, which calculators have been doing without any mistake since a long time.
The point is perhaps confirmed by another comment further down in the post
> by solving hard problems you get an insight into the problem-solving process itself, at least in your area of expertise, in a way that you simply don’t if all you do is read other people’s solutions. One consequence of this is that people who have themselves solved difficult problems are likely to be significantly better at using solving problems with the help of AI, just as very good coders are better at vibe coding than not such good coders
People pay coders to build stuff that they will use to make money and I can happily use an AI to deliver faster and keep being hired. I'm not sure if there is a similar point with math. Again from the post
> suppose that a mathematician solved a major problem by having a long exchange with an LLM in which the mathematician played a useful guiding role but the LLM did all the technical work and had the main ideas. Would we regard that as a major achievement of the mathematician? I don’t think we would.