apple keeps talking about the Neural Engine. Does anything actually use it? Seems like all the current LLM and Stable Diffusion packages (including MLX) use the GPU.
If the energy efficiency of things like Face ID was indeed so far so bad that you need a more efficient M3 Ultra, how come Face ID was integrated into smartphones years ago, apparently without significant negative impact on battery life?
FaceID was just one example they gave (which is probably faster and more energy efficient now).
Image recognition, OCR, AR and more are applications of the NPU that didn’t exist at all on older iPhones because they have would be too intensive for the chips and batteries.
That's false. Face ID is in fact a complex form of image recognition, so image recognition was definitely possible on older NPUs. OCR is the simplest form of image recognition (OCR was literally the first application of LeCun's CNN), so this was definitely possible as well. "AR" is an extremely vague term. If you refer to Snapchat style video overlays, those have been possible for a long time as well.
The original question was asking what features have taken advantage of a NPU. Face ID was introduced with Apple's first "Neural Engine" CPU, the A11 Bionic.
You're confusing this with what features/enhancements new generations of NPUs bring, which nobody else was talking about. Everyone else in the conversation is comparing pre- and post-NPU.
The original question was clearly about the NPU of the currently discussed M3 Ultra, which is twice as large as the previous one. The question is what this one is good for, not what much, much smaller NPUs are good for which have nothing to do with the M3 Ultra topic.
Indeed, but the neural engine does this faster and using heavier models. For example, on-device Siri was not possible until the introduction of the neural engine in 2017.
Historically no, Ollama and the like have only used the CPU+GPU.
That said, there are efforts being made to use the NPU. See: https://github.com/Anemll/Anemll - you can now run small models directly on your Apple Silicon Mac's NPU.
It doesn't give better performance but it's massively more power efficient than using the GPU.
The Neural Engine is useful for a bunch of Apple features, but seems weirdly useless for any LLM stuff... been wondering if they'd address it on any of these upcoming products. AI is so hype right now it seems odd that they have specialised processor that doesn't get used for the kind of AI people are doing. I can see in the latest release:
> Mac Studio is a powerhouse for AI, capable of running large language models (LLMs) with over 600 billion parameters entirely in memory, thanks to its advanced GPU
It will be interesting to see if the original commenter comes back and says what their criticism of Buterin is, but it seems to me that he gets more criticism from cryptocurrency die-hards (especially the more bitcoin-inflected kinds) at this point, because they increasingly see him as an out-group liberal, than from cryptocurrency haters.
As someone closer to that second group, I think he's been the most consistently interesting person in that whole space. He essentially never writes or talks much about the speculative asset side of the industry (which unfortunately comprises like 99.99% of what it is), but rather focuses on the computer science of it, which (IMO) continues to be interesting, even as (IMO) the fruits of that technology have continued to not be very interesting.
