Waiting on simonw "Generate an SVG of a pelican riding a bicycle " benchmark to judge this model

It might be more capable, but AA indicates it's a lot less token efficient than Grok 4.6: https://artificialanalysis.ai/agents/coding-agents?agents=co...

Apart from fictional use cases, what is the real use case here? The pricing on some open models are absurdly low for generic tasks. For the privacy conscious it makes sense to run something like a 8-27B on local network and get the work done.

Are there perhaps some industrial or agri use cases?

Fair, we gotta do a better job at explaining this properly!

So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.

So it will run on those devices, but when/why would you do that?

All of the demo setups (smart home, robot vacuum, watch, etc) could easily have access to a bigger model running on a more capable device either locally or via the internet.

You've explained WHAT it could run ON. You haven't shown any examples of its intended use, from input to result. I have no idea what is expected of me to be entering into that home automation example, and as everyone has handily demonstrated, it's dogshit at what people actually try to feed it so clearly im not the only one.

At my current company we're evaluating small models embedded directly in the web app to provide a natural language interface to the app without spending money on inference (and ideally avoid a ChatChipotle situation where people end up having free token going through our interface).

And needle is one of the most promising model due to its original architecture (but we still need to finish building the actual eval dataset before making out final call).

You can embed much bigger models into web apps with wasm and WebGL or WebGPU. I have a web app running a 0.6B embedding model client-side.

WebGPU is a non-starter for production usage since the support is too limited (No Firefox support, no Linux Support, no Apple x86 support, no MacOS <26).

But yes we are also considering bigger models, though we'll pick the smallest model of sufficient quality because not having to download a 600MB bag of weight is a feature in itself.

Thanks for considering needle. Keep in mind that you can also fine-tune the model to fit your use case more. I think this illustrates the intended deployment pretty well, where both computational resources and compute credits can both be issues for deployment.

We definitely intend to try fine-tuning, don't worry we're not going to dismiss needle just because the base model's performance is too low ;).

Kinda weird to generalize "LLM". Every lab, every model is different. Has its own biases, reward functions etc.

One great use of agentic coding is being able to add and very granular tracing instrumentation to help with these sort of optimizations.

Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.

That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload.

You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything, when you get to the 10-30% range.

The only way you get to “majority” is if your trace implementation is slower than time travel debugging and provides less information, but then why choose something worse in every dimension.

I'm just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I've seen a lot of projects that will resort to manually annotating functions with OTel trace spans, which are ~millions of times more expensive than function calls. Even eBPF uprobe/uretprobe is 100x more expensive than XRay, at a minimum. HotSpot's JFR is like a miracle compared to what people suffer through to diagnose Rust+Tokio.

1: https://llvm.org/docs/XRay.html ... is there even a Rust analog to this?

Not even an analog: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/...

It's worth pointing out though that just tracing function calls isn't good enough for the kinds of stackless coroutines that run in async Rust tasks. You need a way of mapping between the async tasks and the compiler emitted traces.

afaik, C/C++ have the same problem.

The difference is nobody in the C++ community believes that a dominant asynchronous executor library exists, and there is not a pervasive belief that it would be helpful.

The "C++ community", if it even exists, barely believes in sharing code let alone any library being "dominant." They'd have to agree on a build system first, after all.

But honestly that's a mischaracterization of the situation in Rust. Tokio is popular for networked service backends. If that's the wheelhouse you're in then yea it might look "dominant."

You don't need a build system to share code.

You can share with header files and respective (shared) object files regardless of the build system you're using. Likewise you could just share the source. None of this needs a build system.

I was just being a bit sardonic because the C++ ecosystem is so fragmented that something like tokio couldn't really exist. It would be one of three executors in boost, abseil, or folly, and you would never see the kind of downstream ecosystem build on top of them because C++ shops are allergic to external dependencies.

Any organisation that cares about security should be allergic to external dependencies, that is why companies like Nexus and JFrog exist, with companies paying to keep internal repos infrastructure in shape.

One just doesn't install willy nilly from the Internet into the CI/CD pipeline.

Well, they do, and then spend a few late nights when there is a bunch of CVE to fix.

Using an artifactory instance as the origin is not functionally different from installing something 'willy nilly' from the internet. It addresses a narrow range of threats while predominantly being more reliable and faster than public repositories.

It doesn't fix the actual problems with C++, which is that it's significantly more difficult to get and use external dependencies because of the compilation and linkage model of C++ libraries.

If C++ were as easy to build and link as modern programming languages you'd see the same kinds of tools as cargo, and the same kinds of ecosystem evolution as rust, like tokio. But you don't, because C++ code sucks to build, package, distribute, update, and reuse.

(I'm aware/have used conan/meson/vcpkg/etc - doesn't change my opinion).

It surely does, because in most companies that care about security it isn't a mirror, rather the only third party packages that developers are allowed to use beyond the standard library.

Additionally, making new packages available for consumption requires approval from IT and possibly legal, before they become available for consumption.

What is hard is people educated in scripting languages not wanting to learn about toolchains.

The moment Rust depends on other programming languages, we get a build.rs spaghetti file, depending on the knowledge of those writing it, or people throwing away Cargo altogether, and replacing it with Bazel, buck2 and co.

Of course we do, it is done via OS package managers, commercial libraries and SDKs.

More recently, via vcpkg and conan.

Huh, it seems xray puts blank trampolines all over your binary? That sounds pretty nifty but I would expect it to be pretty language agnostic, ish? Adding support should be doable for Rust as well, right? Anyways, pretty nifty.

I am by no means an expert, but I've recently improved performance for some code and used tracy. They have rust bindings as well. It's pretty cool and it seems to be low overhead. Wonder if I can couple it with something like xray? Tracy is more the tracing library + tracing interpretations/aquisition tool.

Edit: apparently rust already supports xray natively on the nightly.

One legitimately great thing about LLMs is that it makes it feasible to add these kind of tracing instrumentations temporarily for profiling and then throw them away so they never reach source control let alone production.

I can get an LLM to trace my incomprehensible Tokio application which was also written by an LLM, which is why I don't understand its behavior. Truly the future we were promised.

I guess you should adopt RFCs or ADRs to help clarify the Tokio application, like this https://github.com/brunoarueira/thoth-mesh/tree/main/docs/ad.... This project is vibe coded, but I had put the effort to create issues, roadmap and ADRs, so later I can understand the project without going deep on the code!

Reaching source control is fine as long as there is a compile time flag to disable the whole thing, which tokio-tracing does

I'm not sure how other people are using LLMs for instrumentation, but IMO the layer you want running in prod is very different from what you want running for a one-off test. E.g., I have some code floating around which burns a pinned core on increasing a counter, with a little wrapper code around grabbing real timestamps at the beginning and end of a session and converting between the two units of time. It's helpful when microbenchmarking a very small unit of code as it actually behaves in a larger program (not perfect -- obviously tweaks the icache and pipeline behavior at a minimum -- but no measurement has zero tradeoffs, and you're always choosing which set of tradeoffs you prefer). An LLM can quickly instrument the call path I care about while I study this or that intervention. The ability to bang out a large amount of throwaway code is delightful.

Was this in a specific application? I wouldn't necessarily expect that histogram to be particularly bad for most applications.

Reading the clock every time you jump into a closure is in fact incredibly wasteful, and is exacerbated by chopping work up into tiny chunks for questionable reasons.

Just curious, why? Is this true even if you did something like a per-CPU histogram that uses atomic ops to increment?

If you have a per-cpu metric there would not be a reason to use atomic instructions to mutate it.

In general your unpinned userspace threads will hit the same CPU 99.99% of the time, but not 100%.

Sure. You get the pointer, you lock the mutex, 99.99% of the time that is uncontended, then you set all the metrics and release it.

Taking the mutex uses (uncontended) atomic ops.

If you’re not using eBPF to trace your app you’re doing it wrong.

Doesn’t that only work on Linux? And then only for things that make syscalls? Presumably people have to trace other slow paths sometime.

The low cost of eBPF tracing is another myth.

1) Its no myth, but you can definitely foot-bullet into doing it wrong, and 2) it's a far better path to take than in-app telemetry.

Lot of this is outdated. Including stuff that made a lot of sense in the past like diataxis and the google guidance.

I've started to build docs structured as a skill. Since every model/harness is trained to handle skills well today. The entire docs of a project or modeled as a skill.

I also have additional frontmatter on the md files. Specifically 2 keys -> when should you read, when you should not read this doc. This with a simple cli to help parse this has made the docs experience much quicker and nicer.

You can also generate svgs, mermaids etc on the fly now when you are really stuck on something.

Referencing Joel Spolsky's article about how you should never do a rewrite is starting to feel outdated as well.

I think it's worth more scrutiny today, rather than less. Your claude code can barf out "a rewrite" but is it any good? So far the answer is "no" (see anthropic's C compiler, or a more recent port of bun).

Software is still the best specification for existing behavior..

I'm not following because a) The bun rewrite was a success b) the C compiler wasn't a rewrite and c) "Software is still the best specification for existing behavior" seems to imply that rewrites are achievable because we already have a working version that functions as a spec?

Why is diataxis outdated; don't your engineers ever read the code? Don't you think it helps the agents too?

Its just far easier to ask an agent to do the first past ( with references to actual code ) and then start to dig in.

A good implementation of something like this is https://deepwiki.com/ for OSS projects.

The code doesn't usually capture the whys; that is what documentation is good for.

Why do you think diataxis and the google guidance is outdated now?

Ah yes, the usual drop down where we need to confirm we are above 18.

Might have to do with a lot of thought leaders and influencers pushing this.

In my own life/career I have seen that over a long enough horizon that lower shame seems to generally work out for people on the long run if you can back it up with some amount of confidence and competence.

I wish folks spend more time on agentic music genertion instead of a single black box prompt to music.

Similar to coding, where you are have a bunch of tools that can do the drudge work for you (the mundane things in mixing mastering for example), but still retain full creative freedom and expression.

I am not articulating this well, but basically transitioning from IDE->Coding harnesses what is the DAW-> ??

"SWE-2 is post-trained from Kimi K3"

Yeah, I'd expect model performance to be super spiky on SWE work, at least they admit it with the name of the model. It's distilled from an already-distilled model.

Maybe still worth it if their "64% cheaper" figure holds.

I don't think you know what distill means

I guess I don't. Does post-training from another (larger) model not fall under the umbrella of distillation? I'd imagine it leads to the same spiky-ness issues...?

Distilling you don't have the actual model weights of the teacher. All you have are the teachers answers to a lot of questions. You then teach your own smaller model to answer more similarly to the big teacher model.

Fine tuning you have the actual model weights of the original model, you then train that model to answer in a different (or better) way.

Distillation requires you to have the actual logits of each token from the teacher model, which in practice means having the model itself.

What you're describing is just synthetic data.

Note Anthropic misused the term in their post about Chinese model distillation, deliberately I assume.

There are multiple kinds of distillation

https://arxiv.org/abs/2106.03310

https://arxiv.org/abs/2207.12106

With the Devin subscription even at the 20$ plan, they offered unlimited SWE 1.7 usage. Wondering if they do the same for SWE 2.

SWE-2 is free for all subscribers on the CLI to try out for the next month :)

What about the gui/windsurf app? Same as cli?

I presume post training is significantly easier than the distillation/training the top Chinese labs are doing.

I wonder if, similar to the American labs, they'll become stingy with their weights once they start getting immediately undercut by a wave of slightly better derived models.

sqlite for everything

NVMe drives + Litestream + object storage(S3/R2..). sqlite simplifies things for the entire long tail of apps/services that aren't the Ubers and AirBNBs of the world.

It looks interesting, but deciding on where to put object storage is what keeps me from doing this. I don’t have an AWS or Cloudflare account and I’m not sure what to commit to.

Also, apparently Litestream could use a filesystem instead of an object store?