FWIW, you're comparing a training-specialized chip to an inference-specialized chip. It'd be more apples to apples to compare to TPU v4 lite, but I can't find that chip's details anywhere beyond some mentions in the TPU v4 paper: https://arxiv.org/abs/2304.01433
How does a training specialized chip function? Forward mode is simple, just a dot product machine. But how do you accelerate backprop on hardware? Does it have the vector Jacobian transformation lookup logic and table baked into hardware?
Mostly you need to be able to stash intermediate products computed in the forward phase so that you can access them in the backward phase. This requires more memory, more memory bandwidth, more transpose, and also, training usually operates at slightly higher precision (bf16 instead of int8 as one example).
I think it's helpful to categorize the things that go into an ML accelerator as those that are big picture architectural - things like memory bandwidth and sizes, support for big operations like transposition, etc., -- and those that are fixed-function optimizations. In all of these systems, there's a compiler that's responsible for taking higher-level things and compiling them down to those low-level operations. And that includes the derivatives used in backprop - they just get mapped to the same plus a few more primitive operations. While there are few more fixed functions you need to add for loss functions and some derivatives, probably the largest difference is that you need to support transpose (and that you need all that extra memory & bandwidth to keep those intermediate products around in order to backprop on them)
Look for the section "CHALLENGES AND OPPORTUNITIES OF BUILDING ML HARDWARE"
But then things change more when you want to start supporting embeddings, so Google's TPUs have included a "sparse core" to separately handle those (the lookup and memory use patterns are drastically different from that of the typical dense matrix operations used for non-embedding layers) since TPUv2: https://arxiv.org/pdf/2304.01433.pdf
Yeah. Sebastian Lague's game videos are great because of the way he walks through his steps & methodology, but wrt his geographic implementations it's not how you would approach this problem for this level of granularity.
I learned how crucial moderation is for a community site like this. I originally hoped that the "good" users would overpower the trolls and create a self-sustaining community without any deliberate moderation. I was very wrong. I knew there were some people on the internet who like drawing penises and swastikas, but I totally underestimated the volume of these people.
I also learned the importance of bringing on additional help. I naively thought I could do everything myself (and I still like flying solo on the development side), but bringing on a volunteer to help moderate the site and manage the community has been incredibly helpful.
Too late for what exactly? Majority of climate scientists agree the situation is dire, but there are many actions that can be taken to reduce the short-term and long-term impact, via both adaptation and reducing carbon emissions.
When climate change was far away enough that we could have solved it with incremental changes, skeptics in the media mocked and denied it.
Now things have got bad enough that we would need more radical changes to keep to 1.5-2 degrees. But radical changes are just the kind of thing that swing voters don’t like, so we probably won’t do them.
I doubt we will come together and implement radical economic changes or geo-engineering solutions. More likely in my opinion is every country for itself, the rise of new, authoritarian governments and 3+ degrees of heating. I would take some kind of corporate tech bro dystopia.
>Majority of climate scientists agree the situation is dire, but there are many actions that can be taken to reduce the short-term and long-term impact, via both adaptation and reducing carbon emissions.
Am I the only one thinking that it reminds me of a bunch of priests in ancient time who claim to speak to the gods and tell us what we shall do?
I'm no climate denier but I hate how everything is framed to induce panic and fear.
Well, there were books and papers warning about the societal, emotional, economic, and environmental impacts of a technocratic society as far back as 50-100 years ago. Don't you think that a little fear and panic, at this point, are justified?
And aren't there enough religious people inducing fear and panic into society still today?
I once heard that you will not meet a more dejected and depressed person than a climate scientist. Imagine warning about things for literally decades or your entire life, being ignored the entire time, and only to be asked in the present "why didn't you warn us?!" or "what's going on?!".
Fear is justified because we honestly don't know about some of these things. Dynamic systems with bifurcations, chaotic behavior, and tipping points are difficult to understand even when they're laid out in front of you, much less when you're trying to figure out what the dynamic system is (in the case of the climate and environment, although we do know a lot). Does it even make sense to continually push the boundaries to see what we can get away with? Do humans realize that this Earth was not made for them and that there were times that humans could not be supported by the Earth's environment? In our daily lives, we stress certain behavior of planning and restraint, but we are incapable of doing so collectively.
Literally the simplest thing in the world to do is to restore lawns and grass areas with native plants (wildflowers, bushes, trees, etc.), and yet no one is doing it. In fact, we're still destroying habitat. And then people write articles like "where are the bees and butterflies?".
I believe earth will survive anyway, and if we die in the process so be it.
Either we find a way to be in harmony with Earth (not consuming more than what it can generate), or we die. Currently we are in a path where the majority of us are going to die, that is self regulation from Earth.
Restoring lawns with native plants is just like sacrificing your elder child to appease Gods, you thing it will help where in fact we just need to go back to the a sustainable equilibrium (which noone knows precisely).
I can’t think of any sentiment that I commonly hear that’s as ghoulish as that. As if the moral value of a human life is somehow less than her lifetime net carbon emissions. My only solace is the hope that people who believe that sort of thing will go extinct and be replaced by people whose values are closer to mine.
We're up against massive loss of biodiversity, loss of habitable space on the planet, and an enormous amount of human suffering and loss of human life due to the current and future effects of climate change. A smaller population will not solve these problems alone, but it will buy us more time to solve them.
FWIW, I'm not trying to promote anything extreme like population control policies, just pointing out that the current trend of population leveling off is generally a good thing.
That seems to be the trend. It makes sense, people who choose a laziness born of pessimism are probably not virile enough to raise children.
Surprised to see this kind of thinking here though; if you are an engineer who is good at solving problems, you should raise children that are also good at solving problems. Point them at our climate/sustainability/energy issues.
I feel like this happened to Gitlab / GitHub as well. Gitlab was gaining a ton of popularity and momentum with their free private repositories and built-in CI testing infrastructure. Then GitHub swooped in and offered all the same things, taking the wind out of Gitlab's sails.
It depends on the goals of the library maintainers. Users who really care about speed may already be using another library such as PDQsort or ips4o. Given that, the stdlib maintainers may reason that adding code and 'complexity' (not all developers understand or are able to maintain SIMD) is not worthwhile.
Conversely, they may prefer to work towards the standard library being the fastest known way of doing things. This is now much more feasible given the single portable implementation, vs. having to rewrite thousands of lines for six instruction sets.
I can't find the documentation for it, but you can see here that they measure the size of the source file after gzip compression, which reduces advantage of code-golf solutions:
Thanks Isaac. How do I reach that page? I expected it on one of the informational pages such as the home page or benchmark programs page, but had no luck.
Not exactly what you're looking for, but here are some 2D plots of code size vs. execution time with geometric means of fastest entries and smallest code size entries of each language:
