Your analogy is poor, and you are missing a very important fact.
Most of the human written code, in places where that code needs to make money, is decidable either entirely or in large parts. I.e without running the code, you can take a domain of inputs and build a complete range of outputs solely by looking at the code.
The way that works in your head is that you are effectively doing a compilation to a logical like structure, which then you can use to infer what the output will be from what the input is, and its a direct mapping that is invertible and separable, so if you know what the output should be, you know what the input is, you can pinpoint the exact location where it breaks. Thats how humans write code.
If thats not clear, imagine a piece of code that splits strings by spaces, deletes the empty strings, and returns the number of words in a string. The fact that you can say that if you want 3 words, there should be maximum 2 sequences of continous spaces between words, is you effectively transpiling that program into a latent space inside your brain neurons and inverting it.
LLMs essentially do this, with the added advantage of having been trained on a HUGE number of codebases, so they can recognize patterns that a human cant.
Where LLMs struggle is complex behavior - they can't simulate things like a human can and choose the best course of action. Even harnesses for agentic loops that can auto run and debug code can't match what a human can do in this regard (hence why self driving still sucks rn).
So moving forward, being a good coder isn't going to be about writing code, or even about prompting LLMs. Its going to be all about whether or not you can design good custom agentic loops, which necessarily involves knowledge of the model at hand (i.e what words you have to use to get it to do the right thing). This will be especially true as investment into "private" inference grows where companies will be using smaller models that have less detailed RL and thus will need much more guidance to do the right thing.
I disagree, to keep it short, what you're describing is not understanding, it's superstition. And I think it's a wrong direction of engineering, relying on some sort of irreproducible expert intuition, one that has been successfully replaced by enlightenment and scientific method.
There are 3 major obstacles in understanding LLMs:
1. They use inscrutable internal language of embeddings
2. They communicate in natural language which is itself ambiguous
3. The weights and training inputs are being hidden as a "trade secret"
"with the added advantage of having been trained on a HUGE number of codebases"
This doesn't really mean much unless we understand what is the quality and relevance of these sources for the problem at hand. Without this understanding it's just a superstition.
>They use inscrutable internal language of embeddings
No different than the electrical signals in the intermediate neurons in your brain that comprises the latent space where all the processing happens
>They communicate in natural language which is itself ambiguous
They lack one-shot precision, sure, but it doesn't matter. They are precise enough with refinement over multiple prompts.
>the weights and training inputs are being hidden as a "trade secret"
For frontier models that make the company money through api pricing, sure. There are plenty of open source models that can be used for the same tasks, which have open weights.
>This doesn't really mean much unless we understand what is the quality and relevance of these sources for the problem at hand.
All of the modern models are RL trained on specific tasks when it comes to coding. I.e the initial training run learns to predict the next token based on context, from all the available texts, but then the RL runs specifically train the model in a harness where it produces code and RLed to produce correct code with specific formatting.
Most of the human written code, in places where that code needs to make money, is decidable either entirely or in large parts. I.e without running the code, you can take a domain of inputs and build a complete range of outputs solely by looking at the code.
The way that works in your head is that you are effectively doing a compilation to a logical like structure, which then you can use to infer what the output will be from what the input is, and its a direct mapping that is invertible and separable, so if you know what the output should be, you know what the input is, you can pinpoint the exact location where it breaks. Thats how humans write code.
If thats not clear, imagine a piece of code that splits strings by spaces, deletes the empty strings, and returns the number of words in a string. The fact that you can say that if you want 3 words, there should be maximum 2 sequences of continous spaces between words, is you effectively transpiling that program into a latent space inside your brain neurons and inverting it.
LLMs essentially do this, with the added advantage of having been trained on a HUGE number of codebases, so they can recognize patterns that a human cant.
Where LLMs struggle is complex behavior - they can't simulate things like a human can and choose the best course of action. Even harnesses for agentic loops that can auto run and debug code can't match what a human can do in this regard (hence why self driving still sucks rn).
So moving forward, being a good coder isn't going to be about writing code, or even about prompting LLMs. Its going to be all about whether or not you can design good custom agentic loops, which necessarily involves knowledge of the model at hand (i.e what words you have to use to get it to do the right thing). This will be especially true as investment into "private" inference grows where companies will be using smaller models that have less detailed RL and thus will need much more guidance to do the right thing.