In scenario 1, I put a Go binary onto a server and make a systemd unit file. In scenario 2, I put a Go binary in a docker container and launch it on a Kubernetes cluster. Scenario 2 is wasting a ton more cycles and RAM, but other than that, what's the difference?
* With containers I can put a Python app right alongside my binary but with total isolation. No need to futz around with chroots or making a static build of Python and embedding my scripts into it.
* What if I need libc, for example to link to SQLite or something? Suddenly my Go binary requires libc. The isolation is broken!
* Systemd can use Cgroups to limit RAM or CPU, but schedulers like Kubernetes can also use your CPU and RAM limits to schedule containers. As far as I know, there's no equivalent tooling for doing this with fat binaries.
* Without Docker, I have to manually manage each container port. Can't run two apps with pprof servers on the same box. With Docker, I only have to care about public ports, and can port-forward into debug ports manually, and with Kubernetes I never have to care about port conflicts.
* Kubernetes, with enough hackery to work around bugs, can actually do seamless deployments where it checks your internal health endpoint or command before making the container available. As far as I know, the alternative to this is writing crappy scripts that try to do this without declarative logic to back it.
You could go on and on. It doesn't have to be Docker. Could use rkt as well, or really any container engine. The point is that the container engine + scheduler pattern is immensely useful, and if it weren't, Google would already be on the next thing.
If you're just setting up a single server with a single program, fine. Drop a binary on it and call it a day. But when you want to implement CI/CD and schedule applications across multiple servers and do load balancing and so on, you might feel like you're reinventing the wheel a bit considering those problems were all already solved by Kubernetes.
The need to configure, manage, monitor and maintain a whole extra set of programs. Also, systemd can just as easily start the binary in a cgroup, eliminating most of the following concerns.
> The isolation is broken
Worth noting that behind the scenes, Linux is using the same in-memory versions of libc, even for containers. That isolation you speak of already doesn't exist.
> [...] scheduling [...]
Ansible, Chef, Saltstack, etc.
> I have to manually manage each container port
You have 65,000 ports available to you. I don't think this is as big an issue as made out to be - people are lazy, which is the only reason there are so many conflicts over 8000 and 8443.
> Kubernetes, with enough hackery to work around bugs
Given the context that I am someone currently attempting that hackery - this shit just kinda works, some of the time. And when it fails, it's a morass of conflicting documentation, and responses of "well, you're not using the default distribution for your environment" (no shit, it's no longer supported) and "works on my single node minikube cluster". Give me ansible/chef/saltstack any day of the week. Hell, I'll even take cfengine at this point.
Kubernetes was built for Google's use case, in Google's environment. Attempting to use it with other restrictions and requirements is a nightmare.
"Given the context that I am someone currently attempting that hackery - this shit just kinda works, some of the time. And when it fails, it's a morass of conflicting documentation, and responses of "well, you're not using the default distribution for your environment" (no shit, it's no longer supported) and "works on my single node minikube cluster"."
You could make pretty much the exact same set of complaints against all those configuration management tools (ansible/chef/salt/cfengine/puppet). They're all a huge mess of spaghetti and hackery that works when they work and can be a nightmare otherwise.
All these tools need at least a couple of decades more to mature.
> Kubernetes was built for Google's use case, in Google's environment. Attempting to use it with other restrictions and requirements is a nightmare.
That's a tremendous overstatement. The first part is just wrong, as was pointed out in earlier responses. The second part just smacks of your having been soured by a poor experience. We started with kubernetes in production at very small scale, just a few supporting services to begin with since there was a lot to learn. Currently we're running five or six of our core production apps on the GKE version of the platform and the experience has been very positive.
>The need to configure, manage, monitor and maintain a whole extra set of programs. Also, systemd can just as easily start the binary in a cgroup, eliminating most of the following concerns.
In the former case, I need to start a (for example) Ubuntu image on my machine or VM. In the latter case, I need to start a Kubernetes image. It's not as different as you think. Kops can do the heavy lifting and even generate terraform manifests, which will put you far ahead of the manual solution in terms of controlling your infra.
Also, Cgroups alone aren't enough, and is only a small part of Docker's isolation, although I did actually acknowledge that Systemd supports them...
>Worth noting that behind the scenes, Linux is using the same in-memory versions of libc, even for containers. That isolation you speak of already doesn't exist.
No it doesn't. Assuming elf binaries, the elf binfmt will call into ld-linux shared object when you hit a dynamic binary, which then does the runtime linking that links libc dynamically. You can confirm that this will call the ld-linux from the container when in a container. Compare how printf acts inside of Alpine containers versus on its Debian host. Or try hosting a CGO_ENABLED=1 Go binary in a container with no libc - it won't load because the kernel won't find the linker. If you put it in an Alpine container and it was built on a Ubuntu host, you might find it cryptically fails to load!
>Ansible, Chef, Saltstack, etc.
This is not scheduling. That is automation, configuration. SaltStack does not calculate how much CPU and RAM is available on a box to balance applications that are running. Before I used Kubernetes, I used SaltStack to manage fat Go binaries and even Docker containers for a moment, so I have a pretty good idea of what the differences are.
>You have 65,000 ports available to you. I don't think this is as big an issue as made out to be - people are lazy, which is the only reason there are so many conflicts over 8000 and 8443.
Nothing to do with laziness. Why would I want to have to pick a new pprof port every single time then remember what app has which? Why would I want to then manage a firewall for all of that? If you're doing this all by hand eventually you're going to screw up. I'd rather have isolated networking deny-by-default.
>Given the context that I am someone currently attempting that hackery - this shit just kinda works, some of the time. And when it fails, it's a morass of conflicting documentation, and responses of "well, you're not using the default distribution for your environment" (no shit, it's no longer supported) and "works on my single node minikube cluster". Give me ansible/chef/saltstack any day of the week. Hell, I'll even take cfengine at this point.
That is an absolutely inaccurate characterization of the very professional and helpful Kubernetes community. I've only had a few run-ins with them, but the experiences were absolutely completely the opposite and I've literally never heard "works on my single node minikube cluster" and I doubt you have either.
>Kubernetes was built for Google's use case, in Google's environment. Attempting to use it with other restrictions and requirements is a nightmare.
And finally, no it wasn't. It was built by the Google Cloud Platform team to help Google Cloud Platform users better utilize their VMs. There's a pretty detailed history that's all publicly available. As far as we know, Google is yet to use Kubernetes itself, other than their spin-off using it for Pokemon Go. Google internally uses some progression of Borg.
We moved our infrastructure from scheduling containers with configuration management to scheduling containers with Nomad... and it is so much better.
We no longer have the concern of defining static ports for each app, configuring App A with the port for App B, autoscaling when traffic increases 3x as someone attempts a L7 DDOS... I could go on!
> schedulers like Kubernetes can also use your CPU and RAM limits to schedule containers. As far as I know, there's no equivalent tooling for doing this with fat binaries.
I much prefer the Docker and Kubernetes world, but you could actually do this (scheduling and bin-packing fat binaries) with Nomad's exec driver:
"Systemd can use Cgroups to limit RAM or CPU, but schedulers like Kubernetes can also use your CPU and RAM limits to schedule containers. As far as I know, there's no equivalent tooling for doing this with fat binaries."
They're called cgroups. You don't need systemd, Docker or Kubernetes to use them.
In scenario 1, I put a Go binary onto a server and make a systemd unit file. In scenario 2, I put a Go binary in a docker container and launch it on a Kubernetes cluster. Scenario 2 is wasting a ton more cycles and RAM, but other than that, what's the difference?
* With containers I can put a Python app right alongside my binary but with total isolation. No need to futz around with chroots or making a static build of Python and embedding my scripts into it.
* What if I need libc, for example to link to SQLite or something? Suddenly my Go binary requires libc. The isolation is broken!
* Systemd can use Cgroups to limit RAM or CPU, but schedulers like Kubernetes can also use your CPU and RAM limits to schedule containers. As far as I know, there's no equivalent tooling for doing this with fat binaries.
* Without Docker, I have to manually manage each container port. Can't run two apps with pprof servers on the same box. With Docker, I only have to care about public ports, and can port-forward into debug ports manually, and with Kubernetes I never have to care about port conflicts.
* Kubernetes, with enough hackery to work around bugs, can actually do seamless deployments where it checks your internal health endpoint or command before making the container available. As far as I know, the alternative to this is writing crappy scripts that try to do this without declarative logic to back it.
You could go on and on. It doesn't have to be Docker. Could use rkt as well, or really any container engine. The point is that the container engine + scheduler pattern is immensely useful, and if it weren't, Google would already be on the next thing.
If you're just setting up a single server with a single program, fine. Drop a binary on it and call it a day. But when you want to implement CI/CD and schedule applications across multiple servers and do load balancing and so on, you might feel like you're reinventing the wheel a bit considering those problems were all already solved by Kubernetes.