I don't think the show-commit or file-at-revision routes are what's causing the bot load.

Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine what commits exist within e.g. a specified range.

I suspect that'd require more work on the frontend than "just run WASM-built git" ... unless the proposal is for it to fetch all requested objects lazily, in which case I think you'd be surprised how many files are read by git when answering a question like "show me the diff by user XYZ in file ABC on branch QRS between date 1 and date 2". That starts to get expensive to pull in the browser, and the bandwidth costs might start to hurt even if the backend now only had to serve cacheable dumb blobs.

This is a much more substantial critique :) I definitely don't think this is a trivial idea. I did a quick analysis and the naivest possible solution isn't very good: only looking at uncompressed objects requires scanning ~340 MB to deliver the <1 MB answer the question "show me the diffs by user Zhang Yi in file fs/ext4/inode.c on branch master in 2025". The bulk of that is scanning commit objects for commits by Zhang Yi in 2025 (171 MB), then fetching trees to filter those to just fs/ext4/inode.c (159 MB).

But this begs the question, why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes? (Making a space-time tradeoff that doesn't make sense on a public-access website in a post-stupid-AI-scraper world?)

> This is a much more substantial critique

Thanks! FWIW, I engaged because I think there's the basis for a good approach here and I want to see if we can expose what works through discussion.

> why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes?

That's the critical question (ugh, as I write this I realize I'm starting to sound like an AI, sorry). It comes down (I think, I don't work on kernel.org) to a few things:

- cgit is ubiquitous and easy to deploy.

- Cgit doesn't have a database, or indexes. It's a fairly thin wrapper over git itself. Git itself is optimized for performance of local filesystem operations, which is nice, but means it can be written with e.g. the page cache in mind for performance, and doesn't need to spend a ton of time thinking about the number of files it accesses--after all, they're all small and locally-available, and thus probably cached.

- Things like cgit often have to combine (the C equivalents of) multiple chained git commands to render certain views.

- Cgit has caching, but it's simultaneously too naïve and too specific to handle use-cases like this. It's too naive in that it's caching content blobs via the filesystem (and using sendfile to serve them) without awareness of e.g. shared cache blocks for deduplication or anything, so it can't cache intermediate states used for rendering e.g. diffs. It's too specific in that it's only caching the results of certain underlying git operations and specified renders, so it can't prioritize e.g. cross-application LRU/frecency patterns when deciding what to cache and what to evict.

In short, what you're proposing is probably best implemented server-side (and hopefully already exists): it's an indexed database of Git's objects that has behavior parity with Git's rendering of the various views/questions people can make of the server. That's a nontrivial undertaking: git doesn't have a VFS layer you could plug a database into the way something like SQLite does; it very strongly prefers procedural file-based accesses to its data. If you could use something like that, you could engineer it like a typical webapp: content-unaware caching layers and indexes to make certain operations fast. Not easy to build, and not easy to swap in in place of cgit (which you also probably have to keep a lot of behavior parity with), though.