Ah, yeah, the CF guys are from another team. They deploy their stuff on (among other things) the VMs created by the OpenStacks running within our Kuberneteses, though. It's turtles all the way down. :)

That's more turtles than I've typically seen. How're you deploying k8s?

Bare-metal with homebrew automation. We install CoreOS via PXE boot, and during the installation it also sets up a Kubelet as an rkt container. The Kubelet then spins up the other k8s components via manifests. The pod and service networks are routed via BGP using our own https://github.com/sapcc/parrot

Later this year, we will go back and evaluate the maturing k8s administration landscape. Our current approach has a few drawbacks, e.g. it requires a CoreOS reinstall to upgrade k8s cleanly (since all the magic happens in cloud-init and Ignition).

Could you email me on my work address? I may have something of interest to share. I'm just not sure if we've made it public yet.