Go Beyond

Only read if you don't mind being offended.


Systemd Insanity

Systemd...

Misunderstood by grey beards everywhere. Easily misused by anyone. And probably easily missed, as well.

For the past couple of years I've been using Systemd pretty extensively. Is it systemd or Systemd? I don't really care.

And I will admit that I have some fondness for it. I'd say a reasonable interest in Systemd matches a bellcurve, but the 0 line is at the middle, not the floor.

First, you hate it. Too much that's too new. It does everything, absolutely everything. It gets into every last concept and has its way. You have to relearn eveything.

  • Oh, ntpd? No, you're going to use systemd-timesycnd.
  • Oh, you manage your network interfaces? No, we'll do that as well.
  • Oh, /etc/fstab? We have a solution for that, too.
  • Oh, sandboxing? We've definitely got you covered.
  • Oh, DNS resolution? Let me take care of you.

And just as you'd expect of something that does everything, there are a lot of bugs. But I'll get to that in a minute. To be fair, you don't have to participate in every extraneous systemd tool. But to use it effectively it ends up being pretty all-encompassing, regardless.

Ascending the bell Systemd interest bell curve

My goodness. DynamicUsers. Oh, there's more! Granular restrictions. Bind mounts voodoo magic! I can do it all!

But wait, there's more! STATEDIR??? You mean I don't have to make stupid user accounts for everything that needs state? Oh wait, not supported in Debian 9.

RuntimeDir! How did I live without you??

Getting to the top of the Systemd interest bell curve

Wow. That's weird. I specified Group=somegroupname and it doesn't work. I have to use the GID for it to work. What is this, 1989?

Well, that systemd-timesyncd thing sounded cool. It just doesn't work and my clocks out of wack, and nothing let me know about it. Never had that happen with ntpd or openntpd.

Hmm, systemd-resolved is an interesting idea. I'll try it out. Oh wait. The daemon just randomly stops serving DNS requests, the service doesn't say it's failed, and there's no errors logged. I restart the service and it works again. Hmmmm. That's kind of important, maybe DNS resolution shouldn't fail randomly.

Descending the Systemd interest bell curve

OH SYSTEMD.

THOU GREAT SYSTEMD.

YOU DO EVERYTHING FOR ME.

BUT YOU DON'T DO X, Y, AND Z?

You'll always, always find another feature that you want your monolithic, do-everything god program to do. And when that day comes and you've forgotten how to play with the Unix Lego, you get pretty bitter about it.

But worse, is finally realizing that this just wasn't meant to be. While Systemd has some massive usability improvements, it is so highly opinionated, so radical, that if you don't drink the Koolaid 110%, it does not work well. And even then, it probably is going to have some issues.

Systemd is not UNIXy. It is not one small, good tool for one job. Systemd is your one-stop-shop, your centrally planned economy. And when they don't have that fairly specific but important thing, it's a lot harder for it to mix well.

Systemd and security

Systemd acts very high and mighty, effectively stating that it is the secure, perfect system. From the secure, append-only log file design (if I recall correctly), to sandboxing options galore, you might think that the basics would be covered.

One practice I've been doing for years is mounting /proc with hidepid=2. This makes it so that you can't snoop and see what every user on the system is doing. Maybe they put a password in their shell. Sure is easier to automate that way.

So you'd think such behavior would be encouraged. Well, turns out that the cgroup mount for systemd (where systemd staus output comes from) is set to always be visible. This means, even with hidepid=2, any low life user account or process that can see /proc can do systemd status and see the entire process tree of your services. It in itself shouldn't normally give out secrets, but sometimes it does and it's just easier that way. In fact, sometimes it ends up being more secure putting the secrets on the command line than files on the disk because the disk can be inspected after and that sercret on the command line, if kept secret with hidepid=2, is only in memory (unless you have swap...) and never touches the disk. So it all depends.

Apparently, I never got the memo.

Sorry, but hidepid= is simply not supported on systemd instances. we require the ability to identify remote processes and most of our services run at minimal privileges, and this means this cannot work. Sorry.

Lo and behold, a heck of a lot of stuff does work in Debian 9 with systemd and hidepid=2. Now it's clearly not intended to work and I'm not sure what's supposed to break, and maybe it is broken in later versions. I don't really want to chance it with something so important. And don't tell Lennart, but I can chmod 700 /sys/fs/cgroup/systemd and nothing seems to explode and my systemd status output is kept secret to everyone but the mighty root. Of course I might also want to make my service files only readable by root, but then I get error messages over and over reminding me of my great sin and how it's pointless to try.

A few threads on this:

I'd say I'm no systemd master but I've used it pretty extensively. You can see some of what I've done with it here if you're curious. I don't know every feature and I certainly don't use them all properly. But I've used a lot of them by now and get to go back and refine what I was doing poorly and iterate.

But now... I've kind of lost trust in Systemd. You can do cool things with it. You can definitely make it work. But I have a growing distrust for it. And a distaste for its massive, monolithic nature. It really does try to do too much. But so, so many good ideas. Many implemented very well. Though truly excessive in so many cases.

I think I will have to go back to raru and other sandboxing methods. I'm also... possibly considering maybe one day switching back over to FreeBSD. Maybe, just maybe. I do think the kernel is more stable than Linux (maybe not enough to matter for most). Lesser hardware support can be a pain. As can elusive user-level bugs, like the Chromium endless tab crashing bug that I think might have just gotten fixed (two years later!).

Regardless, init systems tend to fall into a few categories:

  • My First Linux: Apache, MySQL, OpenSSH. Works fine if you want to run one of a service and never do anything outside of the box. Or if you do, it's just a hack.
  • I WANT MY THREE SECONDS BACK: Parallelized, hacky init systems like whatever Ubuntu and Debian were using for a while. Upstart?
  • Systemd: Could probably make a week long course at least to learn thoroughly and have all of the caveats covered. I guess you could teach it in a day, but then asking how you'd actually use it could take up the other week of testing, debugging, etc.

Speed is not a big deal to me most of the time as far as boot times go. Can be a nice bonus, but often the complexity is unwarranted. But not having to reinvent the wheel with shell scripts galore could be nice. And the whole pid tracking concept from the days of old was ugly. I'd rather have my services managed, logs coming from stderr, and not forking needlessly.

So maybe I'll make some kind of service management system, or set of concepts. Or maybe I'll find something good. I should at least look at what djb did.

Anyways, thanks for reading my rant.

PS: One last, quick note. I want to say that Lennart and the team behind Systemd is pretty darn bright. They've done a lot of work and may well have made something that's a big improvement from the mish-mash of init systems out there. At best, I may have different design philosophy. My ability to execute on it is probably not as good. They are very bright and should be taken seriously. I find that in my uses, it's simultaneously too much and too little. For what it is, I cannot reasonably expect Systemd to be any better than it is. It's a very complex bit of opinionated software, and sometimes rightfully so.