The surgeon is the cousin's father. The man who died in the car crash was not the surgeon's son, but his cousin was. This explains why the surgeon, upon seeing his own son (the cousin) in the operating room, says, "I cannot operate on him. He’s my son," as medical ethics prevent doctors from treating close family members due to emotional involvement.
I’m one of the long term PMC / committers on mesos.
In retrospect I feel this was inevitable to a few key reasons:
* k8s was a second system with all the learnings and experience of building such a system at Google for over a decade. Mesos was birthed by grad students and subsequently evolved into its position at Twitter but the engineers driving the project (myself included) did not have experience building cluster management systems. We learned many things along the way that we would do differently a second time around.
* Mesos was too “batteries not included”: this fragmented the community, made it unapproachable to new users, and led to a lot of redundant effort. Most users just want to run services, jobs and cron jobs, but this was not part of Mesos and you had to choose from the array of ecosystem schedulers (e.g. Aurora, Marathon, Chronos, Singularity, etc) or building something yourself.
* Mesosphere was a VC backed startup and drove the project after Twitter. Being a VC backed startup you need to have a business model that can generate revenue. This led to a lot of tensions and mistrust with users and other vendors. Compare this with Google / k8s, where Google does not need to generate revenue from k8s directly, it can instead invest massive amounts of money and strong engineers on the notion that it will improve Google’s cloud computing business.
Even if k8s hadn’t come along, Mesos was ripe to be disrupted by something that was easier to use out of the box, and that had a benevolent home that could unite vendors and other contributors. Mesos could have perhaps evolved in this direction over time but we'll never know now.
Well said Ben. I am also one of the long term PMC/Committer for the project.
One of the lesson I learnt was that Mesos's two level resource allocations was originally designed for running batch workloads (e.g., spark, mpi, etc.) if you look at the original paper. Use it to run long running services is actually an after thought. We end up finding that we have to do lots of tuning on the first level scheduling algorithm to ensure fairness given that the second level scheduler does not have the full view of the cluster and the first level scheduler does not have enough information to make good decisions. The solution to the problem is actually optimistic offer, which is essentially the k8s model.
Another reason k8s was successful is probably because the golang ecosystem. In mesos, we spent a lot of the energy building basic HTTP layer in C++ due to Mesos's unique threading model. I wish we could have spent those time working on actual useful features.
Thanks for the historical perspective. Might you or anyone else be able to recommend any resources that discuss the efforts to tune the two level scheduler for long-running workloads?
You mentioned:
>"The solution to the problem is actually optimistic offer, which is essentially the k8s model."
Isn't the K8s model more "choose your QoS model" - BestError, Burstable or Guaranteed? Or am I misunderstanding your comment completely?
I was curious about the this:
>"Another reason k8s was successful is probably because the golang ecosystem. In mesos, we spent a lot of the energy building basic HTTP layer in C++ due to Mesos's unique threading model."
Could you say what was unique about the Mesos threading model?
> Isn't the K8s model more "choose your QoS model" - BestError, Burstable or Guaranteed? Or am I misunderstanding your comment completely?
k8s's scheduling model is that scheduler is able to see the entire state of the cluster, thus can optimistically make optimal decisions on scheduling, especially for those long running jobs that are very picky in practice. Although k8s by default only runs the default scheduler, you could in theory run multiple schedulers in parallel (the omega model).
Mesos's pessimistic two level offer model makes it hard for second level scheduler to make optimal decisions because it might not get the right offer it needs. At the same time, first level scheduler lacks application specific information to make the right decision to send the right offer to the second level scheduler, thus the problem. We evaluated many first level scheduling algorithms, and ironically found that "random" first level scheduler sometimes works better than DRF for long running services scheduling.
> Could you say what was unique about the Mesos threading model?
Mesos uses a component called libprocess (think of it as C++ version of erlang). Each actor in the system (mesos master, mesos agent) is single threaded. Thus, all i/o operations need to be non-blocking to not block the actor. This makes it hard to integrate 3rdparty C++ libraries, especially those that involves I/O as they might have a different threading model.
Golang solved this problem using go-routing and bake that into the language. So the golang libraries, especially those involve I/O, are much more composable than C++ IMO.
Refreshing in the sense that it's a look back at what worked and what maybe didn't work so other projects don't make the same mistake. They could have just flipped the code repo to "archived" and moved on without a word.
I think the way you explained your last point really hits the nail on the head in terms of FOSS. I did actually enjoy large parts of Revolution OS, the movie about the creation of GNU and Linux, but the part that stood out to me the most was cmdrtaco explaining open source (paraphrasing here): at the end of the day you end up working on something you need, and then you think “if I need this maybe someone else does to” so you publish the source code and let others use it. This stuck with me because, well, if I publish something I found useful and nobody else finds it useful, oh well. But if they do, that’s really great! I am not saying Google open sourced k8s out of sheer goodness of their hearts, but I think it’s a lot harder to maintain that sensibility when the project is VC backed.
>"k8s was a second system with all the learnings and experience of building such a system at Google for over a decade. Mesos was birthed by grad students and subsequently evolved into its position at Twitter ..."
There's something to be said about how the code is written. I've seen easy to read C++ and hard to read go (I found go's built in http server source code to be a tough read) and vice versa. In Mesos we strive to write very readable code and so I would hope that despite being modern C++, it is approachable for an uninitiated reader.
Keep in mind that actors and futures can be used together, which produces an even more powerful result than either abstraction used alone. We use this programming model in Mesos via a library called libprocess. There will be some talks on this approach in upcoming MesosCons and we'll submit one to Cppcon.
Hi David, can you reveal some of your environment (e.g. on-prem vs cloud), were there any technical reasons for switching or was it primarily a matter of perception of velocity/popularity between the two projects?
Just to add some of my own perception as someone who works on Mesos, Mesos continues to be popular with large technology companies that don't make their technical investments lightly: Twitter, Apple, Netflix, Uber, Yelp, for example. Companies continue to choose a Mesos stack based on its technical merits. The project is still moving fast and adding powerful primitives to support the needs of production environments while distributions like DC/OS are trying to make Mesos more approachable (easy to install, administer) and comprehensive (providing solutions for load balancing, logging, metrics, etc). I hope you will take another look at the Mesos ecosystem at some point, a lot of care has gone into it :)
Not OP, but I think the perception is that Mesos requires you to roll a lot more of the solution yourself. That's fine if you're a large company who can throw hundreds of developers at your platform, less so if you've got 5 or even 50.
The surgeon is the cousin's father. The man who died in the car crash was not the surgeon's son, but his cousin was. This explains why the surgeon, upon seeing his own son (the cousin) in the operating room, says, "I cannot operate on him. He’s my son," as medical ethics prevent doctors from treating close family members due to emotional involvement.