You are correct. In large part, the cost of something like Gemini on a very basic Google AI plan provides far more utility than local LLMs for coding assistance.
There are 2 main reasons for running local LLMS.
1. Process private data/work with uncensored models.
2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs.
The thing that is critical for 2 is that a) you have to have a sweetspot between a pretty good model, which means largest parameter counts, and fast enough token generation where you can run agentic loops. The latter is needed because you aren't going go get the "intelligence" of larger models to form shell commands and run tools to figure stuff out, so the only way around that is to have custom agentic loops to force the model into doing what you want, which results in more text processing.
From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results. Qwen models chase benchmarks, which results in MoE models (thus the A3B in the model, i.e 3 billion parameters are only active during inference). In general, these are good for very specific tasks, but fail to be accurate in considering cross task data, whereas Gemma, being fully active does a much better job. If you only need to do a very specific deterministic task, those models are pretty good.
As an aside though, if your task involves pure text processing (for example take html data, make it into a markdown document), you can also additive train Gemma270M quite easily all on CPU, and on a decent CPU it gets like 50-100 tok/sec, no need for any extra hardware.
The thing with Macs is that while they can run those models and larger models no problem, the tok/sec is very slow. This limits effectively what you can do with the models. On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things. Compared to workflow with larger models, where file generation often takes like <10 seconds, it takes a while to adapt.
The only benefit of using Macs is the price for Mini and cheaper studios. However, once you reach the total cost of about 2.5k (note that the M4 statedin the article us about 2k), building a gfx card rig is the way to go. You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models.
There are 2 main reasons for running local LLMS.
1. Process private data/work with uncensored models.
2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs.
The thing that is critical for 2 is that a) you have to have a sweetspot between a pretty good model, which means largest parameter counts, and fast enough token generation where you can run agentic loops. The latter is needed because you aren't going go get the "intelligence" of larger models to form shell commands and run tools to figure stuff out, so the only way around that is to have custom agentic loops to force the model into doing what you want, which results in more text processing.
From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results. Qwen models chase benchmarks, which results in MoE models (thus the A3B in the model, i.e 3 billion parameters are only active during inference). In general, these are good for very specific tasks, but fail to be accurate in considering cross task data, whereas Gemma, being fully active does a much better job. If you only need to do a very specific deterministic task, those models are pretty good.
As an aside though, if your task involves pure text processing (for example take html data, make it into a markdown document), you can also additive train Gemma270M quite easily all on CPU, and on a decent CPU it gets like 50-100 tok/sec, no need for any extra hardware.
The thing with Macs is that while they can run those models and larger models no problem, the tok/sec is very slow. This limits effectively what you can do with the models. On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things. Compared to workflow with larger models, where file generation often takes like <10 seconds, it takes a while to adapt.
The only benefit of using Macs is the price for Mini and cheaper studios. However, once you reach the total cost of about 2.5k (note that the M4 statedin the article us about 2k), building a gfx card rig is the way to go. You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models.