As an old ex-vmware eng it's kinda sad reading all these articles about broadcom's controlled descent into terrain. A lot of really cool and useful engineering was done in vmware's heyday, and it was often delivered in rather shaky commercial shapes, but for a while the whole thing kinda worked. And broadcom seems to see it as a company that is long past its ability to innovate, extracting as many dollars as possible before its rapidly decreasing value goes to zero.
There was a solid few years that a lot of business types at vmware behaved as though winning was the default, assured state. It took several years for them to see that that was very much not the case. A real lesson in there, for every successful business today.
Another ex-VMware here (9 years on the Workstation team). It's so very sad to see... I grew up there from a career standpoint and I was proud to be there, loved being there, and I just don't recognize it anymore.
I agree with your point about "winning was the default state." I joined during the Diane Greene era and left in the Pat Gelsinger era, spoke to all three CEOs at different points in one-on-ones, and learned a lot about how different people approach the challenges of running a company and listening to employees.
Pre-Broadcom, sort of toward the end of my time there, there was a lot of unintended self-sabotage that didn't need to happen. We lost people that we shouldn't and shot ourselves in the foot for some really dumb reasons, which just a bit more coordination could have prevented. There were some innovative ideas that got lost in all this, and I believe could have fueled VMware's product line and put it in the right positions for where the sector ultimately went. Alas.
It made a lot of money with its core virtualization tech, which gave it excellent market momentum in datacenters. But the ever growing cloud lanscape was a shift that eventually became an all-hands-on-deck (sort of, and all too late) moment, hence the sudden interest in cloud-based offerings.
IMHO, I think it was never going to be an easy shift. But after Pat left for Intel, leadership never seemed to recover.
No single root cause, but I'd say a combination of rapid growth, a changing market with no obvious (at-the-time) answers, some resource issues, and some missteps by some people.
The company began to grow fast in the years after I joined. VMware's early culture was amazing (a template for any company, imho), but it's hard to maintain that with fast growth. Not a unique situation, but I feel it had an impact on how we addressed certain challenges.
ESX was big, and we wanted to figure out the next big thing. There were a few initiatives, some gambles. Reorgs didn't always match and some initiatives got starved, others over-fed.
Let me take the opportunity to say that we did have a lot of freedom in engineering to build and experiment, which I always loved. This was a strength of the company. Personally, I got to figure out and build Unity in our Workstation product (it let you pop windws out of a VM into the host -- SUPER challenging), the whole remote VM infrastructure for Workstation (took 3 years with a major rewrite), WSX (early-days HTML5 client for accessing VMs using just a native browser and a network connection), amongst others.
But the world changed and we didn't really nail this. Case in point, The Cloud. (And later Docker.)
In early Cloud days, we were trying to figure out what that all this really should look like, how we fit in. We had things in the works that didn't get released or got restarted, we had some really useful bits that got spun out. What we should have built was more obvious in retrospect, but not so much at the time.
I think we had a lot of really good people in the management chain, and overall people were trying to do the right things there. But we did have a few high-impact bone-headed decisions (some short-term cost savings at the expense of longer-term strategy in parts of some departments) and some people who shouldn't have had the responsibilities they did (in my opinion). Some of these decisions may have made sense at those levels at the time due to factors they were dealing with that I don't know about, but these did damage.
Bear in mind, it was a big company in a quickly-evolving industry, and it's not always easy to get things right. I would NOT say "corporate greed" or "screwing over customers" or anything like that. Not while I was there. But missteps and hard questions to answer about the future.
All that said, I enjoyed working there. I left to work on Review Board (https://www.reviewboard.org) full time in 2013, and then my old team disbanded in 2016, but much of the team still gets dinner every month, 10 years later. Still sees movies, still hangs out in a Discord. I wouldn't trade working there for anything.
This is what happens when you let business school grads do whatever they want to make money; particularly, gutting the company over the long run without having to make considerations for the people who did the actual value creation.
If you had to award bonds and/or equity and severance to the people you laid off in order to lay them off, you might have to look beyond the quarterly bump you got for treating customers, vendors, and employees like crap.
I got my current job because I could lead the hyper-v to VMware migration project. Just this year I led the VMware to hyper-v migration project. VMware was such a great product (although a bit cumbersome in the VCF environment I managed). It’s really unfortunate it went the way of Broadcom. In comparison Hyper-v feels like a typical Microsoft product. A bunch of different tools created by different teams to do tasks that feel bolted on and afterthoughts. I have to run FCM to manage the nodes? But I run HVM to manage the roles? But there’s no overall resource management like VMware vsphere or proxmox does natively?
I’ve quietly built a proxmox cluster in our environment for semi-prod workloads for when the inevitable windows host has issues.
Corporate IT is kind of a parallel universe to me, so it is interesting to see how widely used these products are and how impactful these types of events are.
There are a few broad categories to building one of these platforms.. core virtualization ("relatively easy"), device model ("medium hard"), storage management ("hard"), management plane ("medium hard").
VMWare popularized x86 core virtualization and defined their own device model because they were the early market leader. But the device model is pretty bizarre and I haven't been particularly impressed by the guest driver quality over the years. VMFS and vSAN seem good for storage, the latter being better than anything open source. Management plane was mediocre, I remember it took a while to move off Flash and fat clients while cloud vendors were showing none of that was necessary from inception, and everything is kind of in a walled garden by design (embedded DB etc).
I had a lot of positive experiences with VMware (ESXi + vCenter). Every time I had a new project, I tried to keep it as simple as possible and it worked wonderfully. SAN, Distributed Switches, Automatic VM migrations to balance CPU usage etc.
VSAN was a weird one until I understood the logic behind it, but the licensing costs kind of brought on-par to buying a SAN and be happy.
> I have to run FCM to manage the nodes? But I run HVM to manage the roles? But there’s no overall resource management like VMware vsphere or proxmox does natively?
Microsoft hides their best stuff in SCVMM, which is another product and another license they want you to purchase. Anytime I've asked someone from MS why I can't X like I can in vCenter the response was usually, "oh, it's in System Center. FCM, HVM are just stripped down. If you need it you should buy System Center."
SCVMM may technically have the thing you're missing from vCenter... but it's no vCenter. It is a grotesque, MMC plug-in-looking application where nothing quite makes sense to someone accustomed to vCenter or Hyper-V / FCM. Maybe if you're into Azure and slap another layer on top w/ Arc you'll have something that... is less icky but still isn't vCenter.
But, hey, The Reg says that VMware is now claiming that vSphere Standard was never really cancelled.
Microsoft has done a lot of shitty things, including letting Windows Server and SCVMM languish to push/force their customers to Azure. I've been to a few Ignites and the feeling I got was Server development was starving for resources while the things we've asked for were being deployed to Azure only.
I don't think SCVMM is as bad as you portrait. I've used both SCVMM and vCenter. vCenter is so far ahead. And SCVMM was getting there until 2022. When they stopped trying.
Windows development in general, sadly, hence why I refer that Microsoft sees Azure as Azure OS.
If you think about it, all hyperscalers are hardly any different from old timesharing OSes, the only big difference is the way we get to log into the platform.
I actually had so many troubles with vmware on windows over the years that once I tried out hyper-v for the first time last year, I somehow have never touched vmware again. Sure vmware has better configs, cleaner interface and some very subtle design advantages. But damn, hyper-v just fucking works and it keeps working across updates. I have a VM that has been running for a year and it is still perfectly doing exactly what it should. I never experienced that with vmware.
> I actually had so many troubles with vmware on windows...
I assume this means that you were using Vmware Workstation? The VDDK is for Vmware vSphere, which -IIRC- runs on Linux and coordinates computers running ESXi, which are where vSphere VMs actually run.
Not that many people are going to be using Vmware Workstation to run a large number of VMs.
I think you're conflating different requirements/usecases.
Sure, a type-1 hypervisor like vSphere is technically better since there's no host OS creating unmanageable contention and if you have guests with higher performance requirements then yeah it's settling for something half-assed.
But in reality, we have some similar shit to a "fax server" that has manually set up weird manufacturer printer settings (local admins tried to be automated/distributed) and old third-party software libraries that "infects" a system globally (2 versions of that library cannot co-exist on a computer).
If that piece of crap can be contained on a VM, as long as it's contained and keeps trucking after a restart/powercycle everyone is happy. What kind of VM container hosts it is really secondary.
(What the software in question is can probably be inferred from the comment, the software is liked by users at the client so it's not going away)
> I think you're conflating different requirements/usecases.
I disagree. protoculture's core statement seems to be:
A lot of people think VMWare Workstation is the pinnacle product of the company and I have NFI how this came to be.
A company's pinnacle product can be completely useless to me, but that doesn't change the fact that the products of theirs that I do use are very simple and unimpressive when compared to it. [0]
Having said that, I expect that a lot of people think Vmware Workstation is Vmware's pinnacle product because they've never heard of vSphere, ESXi, or anything else the company has made. "People have no knowledge of like 95% of what most companies do." is always a solid first guess when trying to work out confusion like protoculture's.
[0] Please spare me the "But how can it be a pinnacle product if it's not useful to you?" retort and ones like it. I've deliberately made a distinction between a product's usefulness to me personally and the sophistication of that product.
Does your company have a ton of thick-client apps? I feel like so many workloads are on the web right now and thick clients are slowly becoming a thing of the past - but maybe in Enterprise there are still stubborn thick client Win32 apps that are mission critical.
If I were starting a company today I would make sure that the majority of my apps were web based. (CRM[netsuite or sf.com], O365/GoogleDocs, workday|gusto, etc.)
Most high-security environments (banks, govt agencies etc) have a large on-prem footprint. Then there's specialised industries like manufacturing, science and engineering who typically have a bunch of embedded systems, often running ancient versions of Windows or something that needs to talk to a local ancient server. And many old companies have tons of legacy apps which are a PITA to migrate to newer infrastructure. So yeah, plenty of folks out there using omprem gear, it's kinda the reason why Broadcom is able to what it's doing - lots of suckers out there who're married yo vmware and can't get a divorce.
I’ll be honest it’s been over 10 years at this point. But we had to virtualize a dying Gateway 2000 that had Windows XP on it. As part of that we needed ISA to keep working. I found some USB ISA adapters and ordered them. It took a lot of messing around with Workstation, the host Windows 7 and drivers on the VM. I can’t remember all the details at this point :(
Couple of things (having lived in this world). The people running this stuff aren't by default stupid (well, not always), and not assuming they are helps with understanding.
First, someone has to pay to migrate apps, and as often as not the resources to do that aren't available. Sometimes you do the math and it doesn't make finacial sense to allocate those resources. And sometimes you just can't move apps. The vendor might not support vm/cloud/web deployment. Or, very common in manufacturing, you have an app that runs a machine and requires a very, very specific hw/sw environment to be supported.
Next, there's that whole regulated thing. We spend a lot of time in front of federal/national regulators explaining how we do things so that we don't screw up your mortgage or whatever. And those regulators take careful notes. So if I've spent the last 10 years explaining how we have this amazing VMWare deployment that ticks all the boxes, they're going to want some very, very detailed explanations of how HP VME or Proxmox or whatever is going to tick all the same boxes at least as good as VMWare did. And we're inviting a lot of scrutiny for a couple of years to make sure we did it right. It's a lot of work.
Sometimes you weigh everything, not just 'it costs less' or 'Broadcom are a bunch of assholes' or 'we don't want bros on HN to think we're suckers', and the answer is legitimately "the divorce isn't worth it".
At my last enterprise gig (financial, ~85k employees), when I inventoried apps for cloud migration, there were around 1500 what I guess you'd call 'thick apps', and about 75% of them were some variation of "stubborn thick client Win32 (or .NET) apps that are mission critical". Mission critical to someone, anyway, and the number of apps with 1 or 2 users was kinda eye opening. Noone was going to fund the effort to rewrite them to be cloud native or web based, so there was a lot of lifting and shifting going on.
Many businesses might do better pushing everything to web apps, but SaaS introduces a whole bunch of other issues in highly regulated worlds like finance. It's not always the right answer.
Because it is made of a lot of open source components glued together in different weird ways. Sure, I would trust KVM over Hyper-V any day, but the management sucks for Proxmox.
If they build their new Datacenter Manager offering into what vCenter offered and abstract away all the ugliness (imagine creating a "distributed" switch on Proxmox with two clicks) I think they will make a killing.
The other thing perhaps worth noting that the entire stack is open source, even Proxmox's 'special sauce' for management. Depending on your organization, if you can solve a problem with Debian (and Qemu) you can solve a problem with Proxmox.
Anyone requiring that kind of support is likely hitting up Red Hat for OpenShift Virtualization (despite what warts that has, Red Hat meets the "organizational need for support part of the RFPs).
If you can solve a problem with Debian and Qemu Proxmox gives you nothing.
Better to just use same automation you use for VMs to just keep fleet of bare Debian + KVM machines.
Proxmox is for when you don't have a bunch of Linux experts on board. And that's also kinda what enterprises want to pay for, removing "liability" of having to keep very competent team.
...the fact that ends up costing them more everywhere is of no concern to middle management
> If you can solve a problem with Debian and Qemu Proxmox gives you nothing.
Proxmox gives you centralized management of a fleet of clustered systems with a very handy GUI (as well as CLI and API) that's used by many thousands of installations, which is a lot better than try to cobble together something yourself IMHO. Just because you have "a bunch of Linux experts on board" doesn't mean they have to 'waste' their time on re-inventing the wheel for management of VMs.
At my last job we had plenty of Linux folks and primarily ran a whole bunch of OpenStack, but we still used Proxmox for standalone management of VMs for 'tier zero' services (e.g., DNS, SMTP, LDAP, NTP) of various sorts that OpenStack itself depended on.
I've lead roughly $20M in cloud migration projects in the last two years away from vmware to cloud services as replacements. Just insane what MBAs are willing to do to a product to squeeze short term profits. Even more insane that this country largely sees these people as a benefit to society and not the vultures they are.
The magic that is opex compared to capex. I'm sure it makes sense to some bean counter somewhere in the company, but for us who manage the resources it's always a loss.
I think that it boils down to that opex is considered speedy, flexible and agile. Whereas capex reflects slow/glacial but also stable.
This is also why lease exists, why we have freelance contracts vs employee contracts for indeterminate amounts of time.
I don't think anyone finds this flavor of activity beneficial. What's beneficial is the underlying system that is sufficiently free and flexible that great things (like VMware to begin with) are possible. As much as we don't like "this" we don't want to put any bureaucrats in power to squash creativity and enterprise.
Much like freedom of speech means someone can speak abhorrently, freedom of enterprise means someone can do bad or dumb business things. But we enjoy the thing in totality.
> I don't think anyone finds this flavor of activity beneficial. What's beneficial is the underlying system that is sufficiently free and flexible that great things (like VMware to begin with) are possible. As much as we don't like "this" we don't want to put any bureaucrats in power to squash creativity and enterprise.
Ultimately, the purpose of a system is what it does.
We've structured our society such that performing this type of activity is extremely lucrative, and financial engineering is by all appearances the endeavor that humans value above all others.
> Just insane what MBAs are willing to do to a product to squeeze short term profits. Even more insane that this country largely sees these people as a benefit to society and not the vultures they are.
I do understand the argument that it can be considered as a benefit for society if companies that have a very incompetent management that falls for this kind of trap of getting locked in by some vendor without having an emergency escape plan available get milked or even get bust. :-)
Well, it's not actually a trap, since it's not the management who pays the price - it is investors, employees and the society at large that do, while the MBAs have already moved on to their next victim.
> VMware to Proxmox migrations are not affected ...
Amusingly - and acknowledging tiny scale and modest automation requirements of home lab compared to enterprise - as I started migrating to Proxmox a year or two ago, the process of shuffling VMs off my ESXi servers and onto Proxmox was surprisingly painless. Effectively mount the ESXi as storage systems, and then 'copy to here'.
IIRC the only small challenges were a conservative choice on target VM's CPU type (Proxmox seems to dislike defaulting to 'host' CPU) and the network / SCSI controller defaults. I assume designed to prioritise robustness over performance.
Proxmox been great for decades. I used it back in 2011 when I was sysadmin for https://sahanafoundation.org/eden/ - we had multiple live instance running for the site, wiki, translation, GIS. Just when kvm was getting support too (not openvz) so it was nice an snappy running on some donated poweredge 2950's (thanks zen.co.uk!)
Deployed for Haiti, Chengdu etc. All ran perfectly.
So the Linux qemu-img tool can still convert VM disk images from .vmdk into formats compatible with all the other hypervisors. Is there something more to the process of VM migration from VMWare than just using this tool (and manually creating the proper virtual hardware profile in the new hypervisor)?
Obviously the types of supported virtual hardware will be different between hypervisor types, but that's usually something that can be dealt with separately.
The problem is how you access the disks that are sitting on a VMware server. VDDK allows you to connect and read them (in raw format, in fact) at a reasonable speed, and also understands sparseness and change block tracking.
I suspect the bigger problem is simply the fact that VDDK workflows are already defined and integrated into migration software packages. Otherwise, there are other options to copy virtual disks from ESXi hosts to other destinations that preserve sparseness and don't require shutting detaching or shutting down hosts using the disks, e.g., cloning a snapshot to an NFS server that supports sparse files.
> Is there something more to the process of VM migration from VMWare than just using this tool (and manually creating the proper virtual hardware profile in the new hypervisor)?
Yes? Do that for the hundreds of VMs, with a minimal downtime, bonus points for a dumb C-level on your neck.
You would quickly learn what "just converting disk images" is not enough for moving between hypervisors.
Just as an example: last year I moved a company with two Proxmox servers to ESXi. It took a 3 months and another server as a staging point. A minuscule in modern times 1TB+ VM took a whole weekend to be moved. Of course it didn't work in that time.
Because it involved a full backup - which wasn't anywhere fast on Proxmox, reinstalling the host, a full restore and then tinkering with the famous "runs everywhere (and not Java!)" boot shenaningans of that flavour of Linux OS.
And that was with the help of a backup-recovery solution which did support Proxmox, mere 6 months before that I would had stuck with an agent P2V migration of a VM to VM - which would had involved way more moving parts (can't P2V/V2V to a hypervisor which is not even installed yet) and more pain.
VMware is still seen as "platinum" level in my org compared to Proxmox, which I guess to a certain degree is true. But the features in use are basic VMs with vMotion. Proxmox provides that out of the box. Granted we've had a couple issues with a corosync wobble on larger clusters which was fixed long-term by putting corosync on a separate set of switches.
So far the Proxmox experience has been pretty flawless. We run a several hundred k8s VMs orchestrated by Rancher and use the dynamic load balancing feature that came with 9.2 so no noisy neighbours. Haven't dabbled too much with Ceph as the VMs are backed with NetApp arrays but I anticipate it to be pretty smooth when we reach that point.
First time I read Ceph and "pretty smooth" in the same sentence, it is really that easy now days? I have always thought it was pretty heavy on operational knowledge.
Can someone exfiltrate the VMWare source so that it can be distributed at a later date once Avago (Broadcom) cease?
A friend of mine said that VMWare was still the best VM platform due to the ecosystem that built around it, so it'd be nice if it could be preserved for later resurrection.
I know time moves on quickly, but it'd be a shame to lose it entirely.
I know this is promoting criminality, but there's something wrong with society if this is deemed criminal.
IIRC the guy who wrote a hypervisor for OpenBSD found severe vulnerabilities in VMware. It doesn't surprise me and I don't think a quasi-legal fork would be a worthwhile avenue.
There is nothing to be saved there. There is nothing "special" in VMWare now aside from having the enterprise support, and frankly any effort would be better spent making KVM/libvirt more "enterprise friendly"
Leaving VMware just got harder? What? NO! It just became more obvious that you've failed if you are still stuck there. (VMware employee #10/11 -- I lost count. First Director of PM).
And I know that sounds like victim blaming. Anybody stuck in this situation has my empathy. But I just don't know how to get it across to some company execs/leaders that they have been needing to get their act together and get off VMware for coming on near 3 years now.
> I am of the school of thought that even if Broadcom gives VMware Standard away free for a year, it would be hard to convince customers that they need to stick with the platform
Yep. They're building the cage around you. Why would you stick around?
I'm kind of glad this is all happening because it provides a nice example to point to when I try to explain why I don't want to be dependent on a vendor for these sorts of things.
For real, it's been the most validating of my use-open-source ethos than any other examples I can think of! For this open source isn't about the cost, it's about the vendor dependence/neutrality, and that's quite important when it becomes essentially a business requirement.
I have very little pity for any of these customers, excluding maybe small businesses that simply don't know any better and lack the expertise but at every company and client I've been at there were at least 1-2 people trying to convince higher ups to not needlessly build core infrastructure on proprietary components and yet most were and are still being ignored.
Sure, if you have legacy Windows-only software that is proven to not work in other environments, you're out of luck and your hands may be tied but all the VMWare shops I've liberated from this slow, ugly and terrible to automate mess had to pay dearly for that.
How many more rug pulls do these people need? Oh right, they're probably in the middle of some Azure migration anyways.
It can take a lot of time to migrate. People who've known about Broadcom's behaviour for some time may still be working towards solving it.
Or, they may have been naive. Small companies especially tend to have people with a more, I don't know, 'social' set of business expectations? 'I wouldn't do it to others so surely they wouldn't do it to me / they'll reverse course when they see how bad it looks' sort of expectations. I don't think we should feel "very little pity" for people like that (sympathy is human): but I'd instead say, I think they may be taken advantage of, but I feel admiration because an expectation of good conduct is something everyone should have and retain, even if it is breached.
When one becomes too comfortable with one vendor way of doing things, you end up paying 10's of millions of dollars in the end. Move the work loads to KVM, and invest in your own employees, and you can migrate off of VMware easily.
Rsync exists. I remember migrating a red hat 3 with a 2.4 kernel to VMware in 2014. Physical to virtual. It’s just files on a disk folks. Rsync, grub-install. And boot.
I'm in the middle of a massive Azure Local migration and noticed the VDDK was no longer available... Kinda funny because I was just telling a coworker how surprised I was that they offered it. Thankfully I saved off my copy but I have no idea how Microsoft is going to handle this going forward with Azure Migrate. Same issue with SUSE I believe
We run a lot of ESXi with vSphere (part of overall VCF solution). It works very well. But I am part of Big Corp. so they don’t mind paying. We are good via warranties etc using what we already purchased. I don’t know that I share the same sentiment that we could have just used ProxMox. We are a Dell heavy shop and there is a new Dell Private Cloud product that is very promising. More like choose your own adventure.
Dell private cloud lock in would be the death of any company. You don't want to be isolated to one hardware vendor. We've told Dell to kick rocks many times for renewals and would not be able to do that if we were in their Cloud, which requires Dell hardware.
It’s not their Cloud, you buy a bunch of PowerEdge R7725 for example and then choose between OpenShift, VCF and one other I can’t remember at the moment.
The problem is you must use Dell hardware. Sure it makes being hypervisor agnostic much easier, but you then become locked into a single hardware vendor that has you by the balls every 3 years when hardware refresh comes along. We've switched between HP and SMC several times when Dell has tried to extort us.
This is the only POSSIBLE rational argument for this approach. 4x price, on 1/2 the customer base is still 2x revenue with less customers to deal with. They want to keep the Big Corp clients that are happy to pay.
Now if it actually works out like this, only time will tell I suppose?
It will work for some time for Broadcom. They are private equity (PE); jack prices and don't care about the quality. Heard the stories from a previous company taken by PE.
Big business never think of the future. Good example is when two companies merged the brought down the cost of poly. Took a couple years for gravitation to the mean to bring up the base cost. They kept thinking it would stay low form them with being so big.
The Big Business that are staying with VMWare really should be tasking to know the cost of replacement with estimated time frame with a known successor. Yet, MB are blinded by staying with a known working tool. Perform the task on a few servers to reduce cost while investing in the education to do so. Big MBs really don't have a clear idea of investing wealth.
We're running a migration from VMware to Red Hat Openshift Virt for one customer right now which is over 150,000 VMs. That's a big one, but five figure number of VMs is very typical.
> People talking here about Proxmox make me laugh.
We have a (what I thought was) large nutanix setup at $dayjob for epic emr... we have like 10-12 sysadmins for ~500 servers (end user apps, and backend infra like Citrix), and even that seems like a lot.
just for reference most enterprise backup products are not affected by this and will have vddk for backups and restores. if your backup product supports cross platform restores likely migration is not blocked for you, and many free / trial editions are likely generous enough for most small - medium shops to migrate on the free offering
if you already pay for an ent backup product check if it can restore from vmware backups to another platform and you’re prolly fine (for now)
> […] could have been using Proxmox all along without any of the licensing and fee nightmares.
As a user of Proxmox, and observer of VMware infrastructure in other silos in some IT orgs I've been in, there is no way that Proxmox was a viable solution 3-5 (or more) years ago.
Hyper-V could have been used in many instances. I know a few places that chose to start moving to Nutanix even before the Broadcam fiasco. OpenStack is useful for many large scale environments (especially with tenant isolation and usage tracking/billing/chargebacks).
Proxmox is good a good solution for SMBs, but (especially pre-9.x) there were a bunch of things that made it much less usable for large(r) scale deployments IMHO.
I too ran a large VDI, server, and scientific computing cluster based on Linux - libvirt in my case. I saved a bunch of money over VMware. When I quit to go to a higher paying job, they had to rip and replace everything with vSphere because they couldn't find anyone willing to work for .edu money with the skills to manage that despite the massive cost savings.
I had asked for vSphere to avoid this problem, and they balked at the licensing costs even in the good ol' days.
> I was working on a pretty big proxmox setup back in 2016...
Define "pretty big". Proxmox officially supports up to 32 nodes in a cluster and the Proxmox folks themselves have "seen clusters of around ~24 nodes in production" (as of 2025) and for "bigger clusters than that, fine-tuning might be necessary":
There are folks talking about running things in the high-40s and low-50s, but that's not officially support. If you have beefy hardware you can fit a lot of VMs on a single hypervisor (I've always found running out of RAM is often the limit).
A single vSphere cluster can have up to 96 nodes (without vSAN), and vCentre can handle multiple clusters (2500 total hosts; 40k VMs). Proxmox recently released Cluster Manager, which helps with feature parity.
I'm a happy user of Proxmox for a number of years (and helped ran OpenStack before that), but I think for large(r) scale deployments (and even smaller ones) there's a good reason which VMware was so popular. (How many folks are actually at that (large(r)) scale is a different question.)
(Also: XCP-ng doesn't seem to get much press. Haven't used it myself though.)
I have professional experience with all listed (both on vendor and user side), other than hyper-v. Nutanix is probably the closest true solution to VMware, although not 1:1. VMware is an excellent piece of tech run by people who DGAF and just want the $.
I like Proxmox a lot, but it is technically and commercially several years behind. There is no moat, and when stuff slides sideways, you need to understand what's going on (e.g a competent sysadmin). That is changing with the recent Proxmox NA announcement, but still not there.
OpenStack is not the same product at all, and is about an order of magnitude more complicated. Much more powerful, but instead of a good sysadmin, you probably want a team (and maybe some core devs).
I emphasize most, not all. Most companies using VMWare even 5 years ago could have achieved similar infrastructure using Proxmox. It's been a viable alternative for years. It's always intrigued me how easily big companies get pulled into licensing nightmares because their execs get bullshitted into believing a sales pitch at a seminar. Anyone who has worked with Oracle's product lineup instinctively knows what I'm talking about.
Anyone larger like that automatically also can afford the staff to simply address any shortcomings in-house and then own it forever, and perfectly tailored, and perfectly customizable at any time forever. There is no missing feature that a few hours or days of developer time doesn't erase.
Oh it's crude on day one? So what? You're still operating and on your own terms, and every week a few more minutes of work gets added as little pain points get addressed by whoever cares, and before you know it it's better than anything you can buy, and the total cost was less than your previous licensing, and you never had to deal with a serial number or license manager or account rep, never had to care one bit about counting cpu cores or client seats or vms or physical hosts or ips or backup copies or spinning up new dev boxes or more production without warning or any of that bs.
The bigger you are, the less you need the convenience of outsourcing, not more.
> The bigger you are, the less you need the convenience of outsourcing, not more.
Quite often the bigger you are the more the higher ups want risk to be mitigated, and so are often glad to cut a cheque to be able to yell at someone over the phone.
The other aspect is that they want fungibility. As a corporation, you use off-the-shelf solutions so that when you do your big layoffs every few years you can rotate in some other cogs who already know how to operate the standard solution. Building in-house means valuing your employees long-term and we can't have any of that going on.
It's been years i played with Proxmox. Back then it wasn't as easy as Vmware Esxi was/still is: Just boot a pretty small os, define a datastore somewhere, just grow disks, migrate them, etc. All done with some clicks of a mouse button in a nice web gui. Maybe i should start playing with Proxmox again?
Tesco are indeed suing Broadcom in the UK, because they bought what was sold as a "perpetual" license. And also complaining to the UK competition regulator.
Most of VMware virtualization code was stolen from kvm and they never were punished. Our laws are a joke, our courts are largely illegitimate and so to is our government.
You are hallucinating and don't have any clue what you are talking about (VMware employee #10/11 -- I lost count, First Director of PM at VMware).
VMwware sure as hell did not invent the hypervisor, and we were damn careful not to ever claim that. The early unique IP was how to enable a hypervisor to use dynamic translation where required to work around non-virtualizable x86 instructions, we also did well with how to package a hypervisor to run on a host/share host devices etc. Over time that early differentiation went away as we helped encourage CPU vendors to include better hardware virtualization support. And the other big thing we had was just a market focus/desire to deliver value/delight nerdy users like ourselves.
Who is the granddaddy of hypervisors? It sure ain't KVM, and it sure ain't VMware. That would be IBM. They had decades of doing hypervisors on the mainframe when we were getting started as a tiny startup. They sure know a lot about hypervisors and had a x86 hypervisor research project in Watson labs. We were humbled when IBM fellow Rich Oehler came to us wanting to partner because he saw how far ahead we were. That was years before KVM even got started. The IBM collaboration, lead to years of great work with IBM, who helped VMwware get going with ESX Server. Although the focus of VMware since founding was always to end up focusing on the data center/server market we could not have done that so well and so fast without IBM and sales and GTM cooperation.
Now good on KVM and others for pushing open hypervisor technology, and any company in this space who actually cares about their users and customers. I hope KVM helps pound VMware/Broadcom into the dirt.
I worked on the VMware monitor team back in 2007-2008, which probably put my employee number way in the 1000s. Ole Agesen, Carl Waldspurger, and other early folks were still there.
Back then "kvm" meant a remote keyboard/video/mouse device. Linux kvm was announced in 2006, and released in 2007, and required hardware virtualization, which was first released with AMD Pacifica in 2006. Back then VMware's binary translation was locked up in patents, so in the free software world we had Xen, colinux, and other hacks.
Ha. For quite a while it was very small team and had to be for that kind of brain surgery, I have no idea how large it finally grew. Founders Mendel Rosenblum and Ed Bugnion in the early days. I remember when we hired Ole, Carl, Keith Adams, and Ganesh Venkitachalam. And we hired Steve Herrod to run the monitor team, and relieve Mendel from doing that. I was not in the loop on hiring those developers, no reason to be. Except I remember interviewing Steve and Diane talking to me before the interview to be clear I was there to help sell joining VMware to Steve. I did not need to be convinced hard to do that :-) I'm sure I'm missing other key early monitor folks but those memories are fading...
As someone who has worked in this space longer than a lot of people currently in industry have been alive, VMware predated KVM by a number of years. VMware Server was released in 2001... KVM was announced in 2006 and entered the Linux kernel mainline in 2007. XenSource was formed and commercialized the Xen project with XenServer in 2006, the first open source Xen Project release was in 2003.
So for all you can say, VMware released a product to market that provided a server-based hypervisor before any of their competitors. They're not the first hypervisor, but they're the first server-based hypervisor for X86.
VMware was also an early leader in desktop PC virtualization. I don't recall anything comparable in the late '90s. Virtual PC existed, but IIRC it was still a Mac-only emulation product.
One of the nice things about vSphere in a lab environment was the ease of moving VMs between Workstation/Fusion and ESXi, and prior to the Broadcom acquisition, pricing for the desktop and entry-level server products was reasonable, including free products, generous evaluations, and perpetual licensing with update-only subscriptions.
There was a solid few years that a lot of business types at vmware behaved as though winning was the default, assured state. It took several years for them to see that that was very much not the case. A real lesson in there, for every successful business today.
reply