- it's okay when Side A goes after Assange (a journalist) for possessing classified material. Also, Side A encourages journalists in certain countries to do exactly what Assange did.
- it's not okay when Side B goes after journalists aligned with Side A
The "people acknowledge my existence, people hold the door for me" is not about them being idiots. It's Scott arguing that women have it easy compared to men (which may or may not be true, feminists will disagree).
I suspect this tendency is not correlated to political leaning in any way, and the suggestion that it is says more about how you want to perceive people of a particular leaning than anything about them.
You write beautifully. I decided to click on your other comments and found the same. Rare combination of high-density, high-impact vocabulary, and yet high-clarity.
Maybe they'll finally find the nuclear device lost on Nanda Devi, that has the potential to - *checks notes* - poison North India (via the glacier that feeds the Ganges).
What's your opinion on a sudden flooding that happened some years ago in that region. I am an Indian so for some days our news were showing only that flooding news. It was sudden and super mssive and some news people suspected that same device or maybe one of the devices being accidentally going off. It was all speculation but the sudden and massive flooding was also unexplained to some extent. There has been several massive flooding in the region recently but all are due to extensive rain and cloud bursts. But one was unexplained in my untrained opinion. I remember it was some huge construction site. Wha they were building now I have forgotten that
The high-power unit had 300 grams of Pu-238 in 1965. Given its 87.7 years half-life, only 187g of Pu-238 remaining. It's very hard to do much damage with this amount of radioactive material.
U-234 is ~3000x less radioactive than Pu-238, so having ~120g of U-234 is negligible.
I really fail to see a problem with these tiny amounts of non-brittle material embedded into a solid case. It's still very dangerous, but it's locally dangerous (meters away), not at the scale of whole countries.
This is fascinating, because from what I've heard, Warren Buffett did not favor tech stocks. Does anyone know what gave Buffett the faith that this company was a real deal?
It was Charlie Munger who became enthusiastic about BYD after learning about it from investor Li Lu, leading him to convince Buffett to make Berkshire Hathaway's $230 million investment in 2008.
I think this was mostly a Munger pet investment, he had an extremely high opinion about the CEO and could see he was delivering on his goals one after another.
Berkshire was never tech investor. They looked for solid manufacturing with good price and potential to scale like manufacturing. Not everything is tech and you can still grow without being tech.
Not cheap, unless that one specific model is going to be used across tens of millions of devices, with no updates, for the physical lifetime of the device.
It's just a re-invention of kernel smoothing. Cosma Shalizi has an excellent write up on this [0].
Once you recognize this it's a wonderful re-framing of what a transformer is doing under the hood: you're effectively learning a bunch of sophisticated kernels (though the FF part) and then applying kernel smoothing in different ways through the attention layers. It makes you realize that Transformers are philosophically much closer to things like Gaussian Processes (which are also just a bunch of kernel manipulation).
Seconding this, the terms "Query" and "Value" are largely arbitrary and meaningless in practice, look at how to implement this in PyTorch and you'll see these are just weight matrices that implement a projection of sorts, and self-attention is always just self_attention(x, x, x) or self_attention(x, x, y) in some cases, where x and y are are outputs from previous layers.
Plus with different forms of attention, e.g. merged attention, and the research into why / how attention mechanisms might actually be working, the whole "they are motivated by key-value stores" thing starts to look really bogus. Really it is that the attention layer allows for modeling correlations and/or multiplicative interactions among a dimension-reduced representation.
>the terms "Query" and "Value" are largely arbitrary and meaningless in practice
This is the most confusing thing about it imo. Those words all mean something but they're just more matrix multiplications. Nothing was being searched for.
Better resources will note the terms are just historical and not really relevant anymore, and just remain a naming convention for self-attention formulas. IMO it is harmful to learning and good pedagogy to say they are anything more than this, especially as we better understand the real thing they are doing is approximating feature-feature correlations / similarity matrices, or perhaps even more generally, just allow for multiplicative interactions (https://openreview.net/forum?id=rylnK6VtDH).
Definitely mostly just a practical thing IMO, especially with modern attention variants (sparse attention, FlashAttention, linear attention, merged attention etc). Not sure it is even hardware scarcity per se / solely, it would just be really expensive in terms of both memory and FLOPs (and not clearly increase model capacity) to use larger matrices.
Also for the specific part where you, in code for encoder-decoder transformers, call the a(x, x, y) function instead of the usual a(x, x, x) attention call (what Alammar calls "encoder-decoder attention" in his diagram just before the "The Decoder Side"), you have different matrix sizes, so dimension reduction is needed to make the matrix multiplications work out nicely too.
I personally don't think implementation is as enlightening as far as really understanding what the model is doing as this statement implies. I had done that many times, but it wasn't until reading about the relationship to kernel methods that it really clicked for me what is really happening under the hood.
Don't get me wrong, implementing attention is still great (and necessary), but even with something as simple as linear regression, implementing it doesn't really give you the entire conceptual model. I do think implementation helps to understand the engineering of these models, but it still requires reflection and study to start to understand conceptually why they are working and what they're really doing (I would, of course, argue I'm still learning about linear models in that regard!)
It starts with the fundamentals of how backpropagation works then advances to building a few simple models and ends with building a GPT-2 clone. It won't taech you everything about AI models but it gives you a solid foundation for branching out.
The most valuable tutorial will be translating from the paper itself. The more hand holding you have in the process, the less you'll be learning conceptually. The pure manipulation of matrices is rather boring and uninformative without some context.
I also think the implementation is more helpful for understanding the engineering work to run these models that getting a deeper mathematical understanding of what the model is doing.
Have you tried asking e.g. Claude to explain it to you? None of the usual resources worked for me, until I had a discussion with Claude where I could ask questions about everything that I didn't get.
In some respects, yes. There is no single human being with a general knowledge as vast as that of a SOTA LLM, or able to speak as many languages. Claude knows about transformers more than enough to explain them to a layperson, elucidating specific points and resolving doubts. As someone who learns more easily by prodding other people's knowledge rather than from static explanations, I find LLMs extremely useful.
tldr: recursively aggregating packing/unpacking 'if else if (functions)/statements' as keyword arguments that (call)/take them themselves as arguments, with their own position shifting according to the number "(weights)" of else if (functions)/statements needed to get all the other arguments into (one of) THE adequate orders. the order changes based on the language, input prompt and context.
if I understand it all correctly.
implemented it in html a while ago and might do it in htmx sometime soon.
transformers are just slutty dictionaries that Papa Roach and kage bunshin no jutsu right away again and again, spawning clones and variations based on requirements, which is why they tend to repeat themselves rather quickly and often. it's got almost nothing to do with languages themselves and requirements and weights amount to playbooks and DEFCON levels
- it's okay when Side A goes after Assange (a journalist) for possessing classified material. Also, Side A encourages journalists in certain countries to do exactly what Assange did.
- it's not okay when Side B goes after journalists aligned with Side A
reply