Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

"Filesystems (eg. Ext4) usually think about whole blocks, but are designed with the eye for smaller files, i.e. files aren't expected to be more than some ten or hundred blocks in size for optimal performance."

Sorry what?

I mean, ext4, as a special case, has some performance issues around multiple writers to a singe file when doing direct io, but I can't think of a single other place where your statement true... and plenty where it's just not ( xfs, jfs, zfs, ntfs, refs )



God, how do you come up with this nonsense? Can you read what you reply to, or do you just write this to show off because you happened to know some vaguely relevant bit of trivia, but actually have no clue about the subject in general?

Yes, the goal of filesystem is to perform best when files are greater than one block and smaller than some couple hundreds of blocks. This is what it's optimized for. Can it deal with smaller or bigger files? -- Yes, but this is beside the point. Filesystem by design are meant for the sizes I mentioned. It's stupid to store information in files much smaller than single block because filesystems store a lot of file metadata per file. If your files are too small, then you start paying exorbitant price for metadata (filesystem don't optimize for storing metadata in bulk for multiple files because such optimization would mean a lot of synchronization when dealing with multiple files in parallel).

Similarly, filesystems, by and large aren't good for dealing with large chunks of data, like, eg. database backups, VM images etc. Typically, a storage system for large objects will try to optimize its performance by larger-than-block compression and deduplication. Neither makes sense when your target chunk size is in single to double digits of blocks -- there won't be enough overlap between different data chunks and you will pay more for decompression of data that you don't want to read, if the granularity of compressed chunks is too big.

And this is not a secret at all... talk to anyone who works on either database or filesystem or object store -- they will tell you exactly this. (I worked on a filesystem, for the record.) This is why these things all co-exist and are filling their respective niches...


Wow, this comment is very rude and completely out of place.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: