The logging is amazing. Transparent compression, easy search, output in JSON if you want to parse logs, timestamps with microseconds if you need them.
No, writing stuff to a file was not fine. It's only fine if all you do is to 'tail' and 'grep' it, but any time you want to do something more advanced, it's a bloody mess.
Why parse a text date, when it started its life as an unix timestamp? Why write a bunch of regular expressions trying to get basic data such as an HTTP response code, when it could be a field in the log that you could just search for? Why deal with parsing while the log may rotate or truncate under you, when the log system could be making your life easier?
I wrote a whole bunch of log parsing stuff in my day, and so I have no nostalgia whatsoever regarding this "just write stuff to a file" approach.
Amazing for you maybe. If you have a slower spinning HDD journalctl -eu <service> takes 10-30 seconds (not an exaggeration). Meanwhile cat or tail -f of a log file is almost instant somehow.
> Why write a bunch of regular expressions trying to get basic data such as an HTTP response code, when it could be a field in the log
I mean, this is a problem which doesn't require systemd to solve, you just need a better log format. Someone could have standardised a line based logging format with fields and then someone else could have written libraries to write it and libraries to parse it.
> Why deal with parsing while the log may rotate or truncate under you
That's a log daemon problem that I haven't found to exist with any log daemon I have used.
> If you have a slower spinning HDD journalctl -eu <service> takes 10-30 seconds (not an exaggeration).
That's because you're asking for a log of a service since logging began, which may well have been months worth of logging. You're saying "give me everything that happened ever to this service in a pager, then scroll down to the bottom".
If you want tail -f, use a -f: 'journalctl -f -u <service>'. That's going to be a lot faster. You can also use '-b' to make it return a log of what happened since boot (worth looking into the manual for the details of that one), or specify a start date with -S, like '-S yesterday'.
> Someone could have standardised a line based logging format
And what were they waiting for? In the end, systemd did it, but it's a pretty recent invention. There was plenty time to get there first.
> That's a log daemon problem that I haven't found to exist with any log daemon I have used.
Because log daemons classically don't care about this at all -- it's a problem for whatever reads the files that the log daemon generates afterwards.
> That's because you're asking for a log of a service since logging began, which may well have been months worth of logging. You're saying "give me everything that happened ever to this service in a pager, then scroll down to the bottom".
journalctl -fu is just as slow. I don't keep months of logs and the amount of logs I do keep easily should fit within a minimal amount of space. Somehow systemd manages to screw this up completely and I really don't think this has anything to do with user error.
> And what were they waiting for? In the end, systemd did it, but it's a pretty recent invention. There was plenty time to get there first.
I don't know what people were waiting for. But I think what systemd did could have been done in a more transparent and decoupled fashion that didn't involve binary logs. Just because they did something right doesn't mean that the way they went about achieving it isn't inherently flawed. I get that logs suck but fixing that shouldn't require me to change my entire system management stack.
> Because log daemons classically don't care about this at all -- it's a problem for whatever reads the files that the log daemon generates afterwards.
The log daemon I use (svlogd) first closes the current file, renames it and then opens a new empty current file. If it finds that there's now too many old logfiles it deletes the oldest one. I don't see how this could possibly break any tool which is reading the file except for cutting the stream short when tailing during a rotation. Moreover I don't think that's a problem since if you need to do some kind of processing of the logs then you should just put that in front of or after the instance of svlogd in the pipeline (svlogd can be asked to forward logs to stderr which then means you can chain it with something else).
journalctl -fu takes over a second on my NAS with the logs on an SSD even.
I fucking hate systemd's logging so much due to this crap, don't make me read docs for every little thing, I just want to cat a damn log file, pipe it through grep and call it a day.
Haha, I wish. Then that file becomes too large.
So you want to truncate the start. However, removing bytes from the start of an open file is a bad idea.
So you open a new log file every day instead. You really want to compress the old log files too, to save space.
So now you can no longer just grep the logs as before. You're now also duplicating code and configuration for every daemon.
So you write a daemon called `logrotate` that does this centrally instead. Into which you then copy the log paths configured for every program on your system and how long the logs should get kept. However programs don't much like it when you move the log files out from under them.
So you patch every program respond to a signal by checking whether it's log files have been moved. Which then breaks for multi-process daemons.
You come up with a whole buffet of ugly hacks to make that work most of the time.
Then you install a program which doesn't come with a logrotate config.
Then you decide you'd really want a more structured way to analyze logs than writing fragile regexes.
Then you want to run something as a non-root user.
Then you have an ephermeral batch job.
Then you want to ship off all of your logs to another location and really don't want to duplicate the log paths yet again.
Then you want to see what happened just before the server crashed last boot.
Then you run two of a daemon and give them different log paths.
Then you want to look at your kernel and application logs together.
Then you want to delete just last month's logs to free up some space.
Also wouldn't it be handy if there was some kind of index to make things faster?
...
> However, removing bytes from the start of an open file is a bad idea.
A digression, but I keep wondering why filesystems never optimized this scenario. Being able to trim the start of the file is really useful in certain cases, and without support options are pretty bad.
It also seems fairly easy to support at the cost of a few bytes in the file entry (a start-of-data index into first sector), and resizing a file by seeking past end is already a thing.
I have never used svlogd but at it looks like it does read logs from it's stdin, which is a solid step forward as it avoids much of the pitfalls caused by applications writing directly into log files.
Collection is just the first part of a log system though of course.
Over the years I have historically had several production system outages due to journald corrupting its own data store, being unable to log, and services hanging due to being unable to log (process stdout buffer full, journald not reading it, so process stuck trying to write output).
It certainly has improved over the years, and it has been a long time since this issue has happened (on one of the systems I manage at least), but I have accrued some pretty deep seated feelings about it at this point due to historical pain.
It's just status quo. How would you feel about introducing having to log things into file after all you needed was stdout?
Btw, you still can do that nothing is stopping you, but I don't think each and every service needs to decide (and let user configure) where to put its logs, how to separate them (maybe by errors, maybe by users/domains) and how to rotate them.
I'm still not a big fan of systemd but it does make some things easier (but why wouldn't they think for a moment how easy "systemctl restoptart foo" will be to type, tab complete and issue different commands one after another)
The nice thing about docker is that you can use the same tooling on lots of operating systems, not just that it wraps some Linux functionality. Sure, it may involve using a VM, depending on the system, but I still get to just run "docker" and "docker-compose" and such, as if it were native. A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases.
Really, though, if I care about isolation and don't want to use docker, I'm gonna reach for FreeBSD + jails anyway, and I don't think systemd's gonna change that for me any time soon. If I want containers with better tooling outside the target OS, I'll use docker. Systemd falls in an awkward middle that I don't have much interest in.
> A systemd solution would need a VM running Linux w/ systemd even on Linux, in some cases.
But that's exactly what happens with docker anyway (e.g.: Docker is still running in the Linux VM when running it from a different operating system). And considering the widespread systemd adoption throughout the Linux ecosystem, it's far more likely for systemd to be already installed in the VM than docker anyway.
Systemd docker-replacement on a non-systemd Linux would need to run in a VM, though.
Systemd-docker-replacement: works only on systemd-Linux. Tooling for any other arrangement, including non-systemd linux, left as an exercise to the user.
Docker: works on Linux—systemd or otherwise. Has tooling to making working on macOS or Windows non-terrible and lets you use (mostly) the same commands as you would on Linux.
I'm saying that cross-platform tooling is what makes Docker Docker. Systemd can't replace it without replacing the tooling, even if it can do the same thing only on systemd-Linux.
> Systemd docker-replacement on a non-systemd Linux would need to run in a VM, though.
Would it? There's no reason this hypothetical systemd-docker-replacement couldn't be architected in a way that would not have a runtime dependency on systemd-the-init-system.
In any case, I'm not even sure if systemd wants to be a docker replacement (although it does seem to pick up more and more container features lately). But there's definitely some overlap between the two projects (in particular around process/service management) but podman is a much more direct competitor to docker than systemd.
Things that are good: simple syntax, easy to use utils
Hidden gems: cgroups, file systems containment, etc. Systemd could completely eliminate the need for stuff like docker by totally isolating a process.