I've been a sysadmin for over a decade, and I love systemd.
The arguments against it appear to be philosophical, not practical, and at no time have I encountered any defect in it. After a brief learning period I came to fully embrace it, and in actual use it makes my life easier in several minor but appreciated ways.
It's understandable that people become used to convention, and are resistant to change. However, most complaints about systemd seem to be appeals to tradition.
I actually have a technical rationale for avoiding systemd, among others.
This is a comment I was supposed to send to chris_wot in the systemd v227 announcement thread, but his posts got nuked by dang.
Here goes:
----------------------
systemd tries to force various system resources into the Unit abstraction, to a rather mixed effect. Some are IPC resources (sockets), others are abstract entities which are driven by a lower-level primitive (services that drive jobs), yet others still claim to wrap resources but really drive util-linux binaries (mounts and swaps). The information hiding/encapsulation is inconsistent, too. I'm not shielded from the dirty details of transactions and the seven different job queuing modes, and when they actually blow up in my face, the debugging context is very scarce.
All of this takes place in an ad-hoc graph engine that I cannot really observe. I can look at approximations of it (systemd-analyze), but I have no way of directly manipulating it, only indirectly through normal operations on system and service state of which the effects I cannot reliably gauge as they undergo systemd's heuristics. Moreover, I cannot checkpoint the graph state to disk, so I could e.g. replay and reboot from a known-good system state manifest. systemd snapshots are ephemeral (in-memory) and moreover do not fully preserve the graph info, anyway.
systemd clearly has an identity crisis between whether it wants to be an eager or lazy service manager. Publicly, it recommends laziness through so-called socket activation, but in practice people mostly tend toward using the dependency system. The dependency system, of course, exists mostly to calculate ordering requirements for parallel startup. This however comes at a cost of non-deterministic ordering, whereby a service can be malignantly reordered, breaking the boot sequence (NFS seems especially prone). This is somewhat inherent to parallelism in general, but is particularly undesirable given parallelism serves no other purpose beyond startup speed optimization (for which process checkpointing I maintain is a better solution).
Between all of this intermediate complexity, I can end up in stuck start/stop jobs, cyclical transactions and dependency loops without knowing the exact origin, because the graph engine doesn't supply me with sufficiently good debugging context.
And the worst part is that most of this is not even a reliability/fault tolerance thing. The Unix process model has no semblance of anything resembling a "transaction", nor is the Unit object versatile enough. It's all heuristics on a private graph engine meant to topologically sort the edges, so as to accommodate systemd's dependency system (which conflates hard service relationships with softer ordering requirements, and is generally inflexible)... yet at the same time you're actually expected to lazy load everything, but barely so in the real world.
Oh, and of course, handling anything related to Unit objects in PID1 creates an unnecessary coupling between global system state and service state in particular, on top of all the intermediate complexity of the graph.
Actually, I deleted them. Dang didn't delete anything. I'm not going to comment further on systemd, evidently it touched a raw nerve for myself as I've read all the commentary and found I strongly disagreed with the technical reasons behind systemd hate: I guess I was too overly aggressive as I'm going through a rough time right now (depression and suicidal thoughts), so I thought it best to bow out entirely as I wasn't helping the discussion and caused more heat than light.
P.S. Interesting insights though, you make a few fair points. When I'm feeling better I might ponder some more on them. Peace.
Though you seem to want systemd to start things in the same order each time and have it be predictable, that's pretty impossible. Systemd works more on the basis of "this happened; what should be done now?".
Seems adding something to record how it started things and why it did so would help quite a bit. There's some debug mode where it limits how much it starts at once. Then some way to visualize this easily (not just a GUI/picture).
Whoa, please don't drop things like that casually in the threads. People are quick to believe them, but what you said is false. I didn't touch chris_wot's comments.
We never delete comments outright unless the author asks us to, and only kill individual comments (i.e. mark them [dead]) if we are banning a user for breaking the site guidelines, which was not the case here.
As a sysadmin, I find that every time I rail against systemd here, I get constantly challenged by developers who know far more about, well developing, than I do.
Is it so hard to avoid change for change's sake? I'm not opposed to learning new things, but we're all really busy, and when the standard way of accomplishing things changes in every major release (which you're usually trying to deploy to stay on top of the latest developer demand, or audit requirements or whatever) just adds a massive learning curve. Now I've got to learn something new, rewrite a bunch of tools, all because.. why?
There will always be busy sysadmins; there will also always be new sysadmins. If we agree that systemd is a good thing, the best thing to do is to switch over quickly and bite the bullet on the re-learning problem. I understand why it happened that way and there were excellent technical reasons for it, but Debian's and Ubuntu's slow transitions (through insserv and Upstart + logind, respectively) were probably a disservice in the end, if you somehow magically knew that systemd would be the end state for both distros.
I don't expect it to change again in the next 10-15 years, except potentially to a competitor that is designed very similarly but just plain better.
I would like to see more empathy for the fact that systemd is a complex, underdocumented system. Lennart's blog series is great for the "why," but very sparse on "how" and especially "how do I fix it".
> ... adds a massive learning curve. Now I've got to learn something new, rewrite a bunch of tools, all because.. why?
As a developer I go through that "new tech" cycle every 2-3 yrs. Why? 'couse world's evolving. Some evolution cycles are not necessarily good, but we have to steer the ship forward. It's not always nice, but I don't see the other way...
> I am still to find a sys admin that likes systemd.
Here's another one: I spent 18 mos in Google Site Reliability Engineering (and did a lot of pager work at startups before that) prior to transferring to software engineering, I think I can speak to both perspectives.
Systemd is incredible and is a huge advance in maintaining linux systems. It boggles my mind that anyone could compare it to what we were all doing before and walk away saying: "nah, I'll stick with the stringly-typed shell scripts and the stale pid files, thanks."
Specific technical reasons in favor /for sysadmins/:
- unit files are rigidly structured data and this is a Good Thing. Configuration is not code. Unit files are also very easy to write.
- systemd understands cgroups and manages them properly. Every single critique I've seen of systemd is from someone who either doesn't understand cgroups or doesn't realize why they are so useful and important. Cgroups manage jailing, process isolation, resource control (CPU shares, memory limits, block io shares, etc), privilege separation and dropping, etc., and this is all configurable by systemd using simple and declarative unit file directives. Like it or not, we are moving into a containerized world.
- machinectl is already useful for managing VMs and will only become more so as it matures.
- people shit on the systemd journal, but, I'm sorry -- you're wrong about this too. Structured binary logging is simply more efficient for this problem, both on the write and on the query side. The journal also can integrate with the systemd journals running in any VMs or containers running on your system, so you can get a unified and metadata-tagged logging stream of everything happening on the host
- socket activation enables zero-downtime restart/deployment for network services, any incoming connections to the port will be added to the listen queue by systemd before the target system comes back online
I've managed Unix and Linux systems for over 10 years. I prefer systemd based systems where possible. Hit me up offline (email is in the resume linked in my profile) for technical reasons if you care.
Unit files are simple to write, and make incorporating new custom processes even simpler (does it log to stdout? It'll log to journald and then syslog perfectly fine).
Process management is built in and sophisticated (restarts, timeouts, conditions, limits etc.).
The dependency system is amazing for ensuring your system fails sensibly: I can link mountpoints to processes, and easily guarantee that a mail daemon only runs when its network mount is ready, and then make it also shutdown if that mount goes away for any reason. This is huge for self-documenting infrastructure - you can ensure that even manual intervention in the system won't leave it in a weird state, and the why is all right there.
After 20+ years working with unix, I don't mind systemd. Every init system I've used has its quirks and they all get the job done.
Overall I'll hire the sysadmin who knows the most init systems still found in common linux and unix systems and is the most ready to adapt to whichever trend wins domination. Evolve or die.
As a sys admin type, I think it's fine. I prefer it over sysV init, upstart, and launchd. There are other options, but I prefer to stick with the more commonly used stuff.
It took a while to get used to, but it does what I need very well (dependency resolution, process supervision, etc).
"Ok" with sysvinit, sort-of "like" systemd and "hate" supervisor.
Btw: almost nobody considers the point about supervisor. A lot of clients' systems I manage are sysvinit or upstart systems, which use the init system to start supervisord and then use supervisor to (poorly) manage some services.
So even if I get why someone might like sysvinit better than systemd, I have no idea how someone can like sysvinit+supervisor better than systemd.
On the other hand, developers seem to enjoy it quite a bit.
That's what happens when you have developers developing for themselves, not for sys admins (the end users of their work).