I know infinitely less about caching and VM than the author of the linked article, but I was surprised by this part:
Varnish also only has a single file on the disk whereas
squid puts one object in its own separate file. The HTTP
objects are not needed as filesystem objects, so there is
no point in wasting time in the filesystem name space
(directories, filenames and all that) for each object, all
we need to have in Varnish is a pointer into virtual
memory and a length, the kernel does the rest.
I've had more than one systems person give me the opposite advice, that yes, using the OS's caching layer to do your disk/RAM balancing is good, but you should write into files that are divided on logical boundaries that correlate with how you use the data. Their argument was that this gives the caching layer more information, e.g. it can consolidate all your tiny objects into one part of the cache to avoid your small objects unnecessarily pinning a ton of VM pages, and can do things like prefetch pages when you start to read a big object, or even choose not to load a very large object into the cache at all if you're reading it sequentially (keeping it from clobbering the cache). When evicting pages it can also take small-versus-big-object and these-pages-go-together issues into account, as opposed to all pages looking alike.
That's all hearsay, though, and I have no idea if it actually improves things in practice on current OSs or with which kinds of workloads.
Both arguments might be correct. If you are dealing with a language that makes a strong distinction between the object and the memory layout of that object, you might be better off handling the serialization yourself. But Varnish is in C, and data on disk is mapped to memory then used directly as a struct.
Thus there are no small objects scattered around--- each 'object' is contiguous, and most likely on a single page. Prefetches happen automatically --- in Linux at least, disk caching and the VM are essentially synonymous. The benefit of using the VM directly with mmap() is that you have more control over the details and less overhead.
> in Linux at least, disk caching and the VM are essentially synonymous
Ah that could explain it. The people I know seem to be working on fs-level caches that operate at least in part on file granularity, so maybe they assume Linux/FBSD do fs-level stuff as well. They seem to try to do things like deciding whether to cache or not based in part on how big the file being read is, and what its historical usage patterns are.
In this kind of case I find it helpful to distinguish between what optimizations the system could implement and what it does implement. There are so many theoretical optimizations that don't work in practice that you can spend forever debating them (especially since no one can be proven wrong in a theoretical debate). I actually like the abstraction to leak a little when it comes to performance. And keep in mind that PHK develops the FreeBSD kernel that he's using, so he knows what it's going to do.
Varnish tries to limit the filesystem overhead as much as it can.
By only having the one file so the number of system calls for a read or a write is '1', no need to juggle file descriptors (which you really really need for your sockets, not for the file system in most varnish setups).
Having 'small' objects clustered together is a good idea anyway, but you can do that in a big file just as easily as you could do it in multiple smaller files.
That's all hearsay, though, and I have no idea if it actually improves things in practice on current OSs or with which kinds of workloads.