most people in ML have no idea what transformers actually are.
Traditional networks, at every layer, used to be output = [weights matrix][input], where input is a vector, and weights matrix is the weights, where each row corresponds to the set of weights for each neuron.
Transformers upscale the dimension of the data. Instead of the above, transformers do [output] = [input][weights_matrix]. When you multiply an input by a matrix, you get an output matrix back. Thats all that happens. Nothing fancy. You have weights matricies for K/Q/V, which when post multiplied with the input, give you the KQV vectors, and then you just simply multiply them together and apply a scaling factor.
There is nothing magical about K/Q/V. There is nothing about any one doing any querying or any one representing some keys. The naming is just a carry over from how they that selection process is used in pre llm data science fields where you manually define the key and query matricies to define relationships between components.
The reason of why it works is because is an extension of something called kernel tricks from pre LLM machine learning days - you map a lower dimensional space to an extra dimension based on some equation, and it lets you apply some classifier on the combination of existing values and new value. Thats what transformers are doing - they are mapping the individual token to the dk x n_heads latent space, which allows for a higher dimensional representation of the data, capturing complex relationships.
You can do Transformers with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on. The thing is, there really isn't any way to tell if any of that gives you more advantage - it certainly would give you more granularity, but as of right now, in terms of training to generate a specific token given previous ones before it, it seems that you don't need any more dimentions than dk x n_heads. Interestingly enough, you also can mathematically represent any such transformer including the starting one with a sequence of linear layers like in traditional networks, the only thing is that it becomes computationally inefficient due to having duplicates of data.
The reason why RNNs and others and others didn't work is because RNN training is effectively trying to linearly regress on chaotic effects - i.e what set of starting conditions would evolve with a given process into what you want. This is an NP hard problem, and you can't really do it linearly.
Transformer models on the other hand, use breadth instead of compute to capture interactions. In those learned weight matrices, you have a latent space of a bunch of "knowledge" compressed, and an algorithm to search on that "knowledge".
But, its very possible that an RNN can be smarter than a frontier model while being much smaller in size - in the same way that its very possible that you can have the right set of prompts for an existing local inference smaller model that can basically be very close to AGI in terms of being able to solve any problem across any domain. Right now, the space is about exploring those prompts, which is the frameworks and harnesses, to get to there, as well as making the compute portion more efficient so you can explore that space faster.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
> You can do this with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on
As a outsider I have many dumb quesion like these. I am trying to understand transformers in a Occam's razor way. It's a complicated machinary after all.
Hey I like your exposition of attention in terms of the kernel trick, but the big huge difference is that kernel methods use the inner product which is a commutative operation -- it's bidirectional (and both tokens are projected into the space by the same function prior to being dot-producted).
This means that it can't capture unidirectional relationships, like "ball" is the object on which the verb "threw" acts in the sentence "I threw the ball". This relationship is true in only one direction; it isn't true to say "threw" is the object on which the verb "ball" acts.
I do think it would be fair to say that transformers generalize the kernel trick to noncommutative relations by applying a different projection function (W^Q and W^K) to the two tokens being considered. This makes the overall operation (project then dot product) a noncommutative operation.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
Nirvana? Singularity? Paperclips? Vernor Vinge rising from the dead? I'm curious; please share!
The better solution for security is sandbox execution. Most agents used in practice, if given terminal access, can find ways to get around most of the MCP restrictions.
You can verify this yourself. Get an old laptop to act as a wifi hotspot and forward traffic over usb ethernet adapter to your actual router. Then run tcpdump on the computer. You will see the multitude of phone-home traffic.
Is there evidence of this anywhere on the internet? I couldn't find anything with ten minutes of searching, and I don't use little snitch, and I'm not going to put the effort in to remediate this just to figure out if this comment has weight.
> As it turned out, this behavior is on purpose. There’s an explicit whitelist that allows certain macOS services to bypass any third party firewalls and to communicate on the Internet without being even noticed by the user. A hole in the wall.
Was remediated later, but shows there is precedent.
What is "root" in macOS is more or less a power user role. Apple took away the true ability to operate as root a long time ago with System Integrity Protection which walls off the critical parts of the OS from the user entirely.
Unfortunately, SIP also restricts basic system functions that are trivial in other OSes. Apple made this very difficult, and MS would have loved to do this in Vista had they not received the backlash that they did.
You can disable SIP if you want. You can run whatever you want on your machine. Disabling SIP is indeed a security risk… then again, so is running something as root. Having a Mac w/o SIP is no more a security risk than having a Linux w/o a r/o kernel AFAIK.
The way apple talks about SIP its like a miracle we survived at all pre SIP. Pretty sure I've had SIP disabled for years now when I got nagged for something or whatever and had to disable it.
SIP is a good thing and prevents a lot of (usually PEBKAC) problems. It is also the direction where Linux is going (is now? or maybe they gave up? idk, it’s been a while since I heard about immutable distros).
For tech-aware people, it is probably not necessary indeed.
However, given that now “tech-aware” people are running completely random and unvalidated scripts w/o second thoughts (or even first thoughts actually) on their main machine, I’d say the “tech-aware” line is very very high…
Well, I block all connections to iCloud for every app (user and system) completely with Little Snitch, and the fact that it does indeed break some little features I can absolutely live without, indicates that it has at least some effect.
But still, I fully share the sentiment that creators of an OS are perfectly capable of bypassing whatever there is running on top of it.
When you are doing matrix math, compute is compute. Apple cant be more efficient due to physics. The only reason Macs are more efficient in general is that they have tightly bundled hw and sw for specific tasks.
Ive essentially followed that paradigm with Python and C. I start out writing Python code. If I need something to run fast, I build a standalone C application that either reads from a file or listens on a socket, and just invoke it from Python. No need to write the entire thing in Rust and deal with all its semantics when it will be at best like 2% faster.
Write python code, ask any llm to translate it to C, then compile the C code - if it produces errors or fails to run, ask LLM to fix it. Then take it a step further and ask it produce machine code, and repeat the procedure.
Then RL the llm on the above, and you basically have a Python -> Machine code compiler. If you cover every single possible python syntax, every single possible C syntax, every possible standard library call, and all the compiler optimization examples (all of which is a final set), you should get something that is extremely accurate.
Your analogy is poor, and you are missing a very important fact.
Most of the human written code, in places where that code needs to make money, is decidable either entirely or in large parts. I.e without running the code, you can take a domain of inputs and build a complete range of outputs solely by looking at the code.
The way that works in your head is that you are effectively doing a compilation to a logical like structure, which then you can use to infer what the output will be from what the input is, and its a direct mapping that is invertible and separable, so if you know what the output should be, you know what the input is, you can pinpoint the exact location where it breaks. Thats how humans write code.
If thats not clear, imagine a piece of code that splits strings by spaces, deletes the empty strings, and returns the number of words in a string. The fact that you can say that if you want 3 words, there should be maximum 2 sequences of continous spaces between words, is you effectively transpiling that program into a latent space inside your brain neurons and inverting it.
LLMs essentially do this, with the added advantage of having been trained on a HUGE number of codebases, so they can recognize patterns that a human cant.
Where LLMs struggle is complex behavior - they can't simulate things like a human can and choose the best course of action. Even harnesses for agentic loops that can auto run and debug code can't match what a human can do in this regard (hence why self driving still sucks rn).
So moving forward, being a good coder isn't going to be about writing code, or even about prompting LLMs. Its going to be all about whether or not you can design good custom agentic loops, which necessarily involves knowledge of the model at hand (i.e what words you have to use to get it to do the right thing). This will be especially true as investment into "private" inference grows where companies will be using smaller models that have less detailed RL and thus will need much more guidance to do the right thing.
I disagree, to keep it short, what you're describing is not understanding, it's superstition. And I think it's a wrong direction of engineering, relying on some sort of irreproducible expert intuition, one that has been successfully replaced by enlightenment and scientific method.
There are 3 major obstacles in understanding LLMs:
1. They use inscrutable internal language of embeddings
2. They communicate in natural language which is itself ambiguous
3. The weights and training inputs are being hidden as a "trade secret"
"with the added advantage of having been trained on a HUGE number of codebases"
This doesn't really mean much unless we understand what is the quality and relevance of these sources for the problem at hand. Without this understanding it's just a superstition.
>They use inscrutable internal language of embeddings
No different than the electrical signals in the intermediate neurons in your brain that comprises the latent space where all the processing happens
>They communicate in natural language which is itself ambiguous
They lack one-shot precision, sure, but it doesn't matter. They are precise enough with refinement over multiple prompts.
>the weights and training inputs are being hidden as a "trade secret"
For frontier models that make the company money through api pricing, sure. There are plenty of open source models that can be used for the same tasks, which have open weights.
>This doesn't really mean much unless we understand what is the quality and relevance of these sources for the problem at hand.
All of the modern models are RL trained on specific tasks when it comes to coding. I.e the initial training run learns to predict the next token based on context, from all the available texts, but then the RL runs specifically train the model in a harness where it produces code and RLed to produce correct code with specific formatting.
The difference is, the total cost for a human to write/modify code files is much higher than for an LLM. A lot of time has been spent designing program languages so that human time is more optimized. LLMS dgaf if the code is structured cleanly or is a mess.
So even if you end up with a mess of a codebase, as long as you define your test cases and they all pass, what is in the middle doesn't really matter.
most people in ML have no idea what transformers actually are.
Traditional networks, at every layer, used to be output = [weights matrix][input], where input is a vector, and weights matrix is the weights, where each row corresponds to the set of weights for each neuron.
Transformers upscale the dimension of the data. Instead of the above, transformers do [output] = [input][weights_matrix]. When you multiply an input by a matrix, you get an output matrix back. Thats all that happens. Nothing fancy. You have weights matricies for K/Q/V, which when post multiplied with the input, give you the KQV vectors, and then you just simply multiply them together and apply a scaling factor.
There is nothing magical about K/Q/V. There is nothing about any one doing any querying or any one representing some keys. The naming is just a carry over from how they that selection process is used in pre llm data science fields where you manually define the key and query matricies to define relationships between components.
The reason of why it works is because is an extension of something called kernel tricks from pre LLM machine learning days - you map a lower dimensional space to an extra dimension based on some equation, and it lets you apply some classifier on the combination of existing values and new value. Thats what transformers are doing - they are mapping the individual token to the dk x n_heads latent space, which allows for a higher dimensional representation of the data, capturing complex relationships.
You can do Transformers with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on. The thing is, there really isn't any way to tell if any of that gives you more advantage - it certainly would give you more granularity, but as of right now, in terms of training to generate a specific token given previous ones before it, it seems that you don't need any more dimentions than dk x n_heads. Interestingly enough, you also can mathematically represent any such transformer including the starting one with a sequence of linear layers like in traditional networks, the only thing is that it becomes computationally inefficient due to having duplicates of data.
The reason why RNNs and others and others didn't work is because RNN training is effectively trying to linearly regress on chaotic effects - i.e what set of starting conditions would evolve with a given process into what you want. This is an NP hard problem, and you can't really do it linearly.
Transformer models on the other hand, use breadth instead of compute to capture interactions. In those learned weight matrices, you have a latent space of a bunch of "knowledge" compressed, and an algorithm to search on that "knowledge".
But, its very possible that an RNN can be smarter than a frontier model while being much smaller in size - in the same way that its very possible that you can have the right set of prompts for an existing local inference smaller model that can basically be very close to AGI in terms of being able to solve any problem across any domain. Right now, the space is about exploring those prompts, which is the frameworks and harnesses, to get to there, as well as making the compute portion more efficient so you can explore that space faster.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
reply