Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
NASA secures contract with Rocky Enterprise Linux (sam.gov)
196 points by InitEnabler on June 21, 2023 | hide | past | favorite | 165 comments


Looking at the "statement of work", unless I'm misunderstanding - it's 3 user (workstation?) licenses?

To me, it's more like 3 engineers at NASA are interested in it and got their boss to get them 3 licenses? As opposed to "NASA moves all RHEL licenses to Rocky after Redhat destroys CentOS community"?

Edit: I know nothing of how govournment agencies (and certainly not US ones) work.. I assume all purchases go through some process where they're all made public?


Here's how it works (POV NASA/JPL PM of 8 years):

* The institution has a slush / IT fund that provides basic windows laptops or MACs, right off the shelf with lots of security add-ons

* For any non standard equipment or software, the engineers on a project must seek money from the project's budget and justify that cost to the project manager. (not their long-term boss).

* Project managers have a fixed budget based on the proposal they wrote to the funding agency, usually US GOV, with a fixed cost for these kinds of purchases, agreed upon at the time the proposal was funded

* The limited project funds are used to purchase whatever is needed to satisfy the project's objectives. If the project's objectives are not satisfied, the funding agency can stop, or even pull back funding from the project manager

* The approvals for equipment purchases are, at least in theory, given based on merit, which is very often weighed using prior successful proposals. New types of purchases are scrutinized more. If it's traditional to buy CENTOS, then buying ROCKY would be more difficult.

* That precedent is key. Once it becomes clear that ROCKY is being used by successful teams, then the approval process is much easier.

That's why this is a big-ish deal (not huge, but nice to see). Now, people writing proposals and working on projects can point to this as precedent. The best outcome is when funding agencies see "ROCKY is better and more cost effective for the teams we've been funding", then you might get a funder to request that teams use ROCKY as part of the proposal requirements. This is almost nonexistent in federal contracts, but is quite common when dealing with military or commercial contracts.

That, and ROCKY can probably now have a NASA logo on their website under "customers", which adds a bunch of credibility to the project, honestly, whether it's necessarily deserved or not based on 3 desktops.


That sounds a bit more rigid than what I knew to be the case. I think every one I worked with from Goddard had a mac - unless - they specifically asked for something else. That includes both the civil servants and contractors.

Purchases were hard because acquisition was hard but I don’t think the bar requires previous use. Usually getting a vendor to accept a PO was the hard part IIRC


Yeah, as I mentioned below, "Standard" equipment did include macs, and I had a mac myself. I'll edit for clarity.


I've been hearing lately that "buy what you can, build what you must" is the approach to procurement for orgs like NASA and SDA. If it is the case, one would hope this means more folks like ROCKY can get over that initial hump and grow within those orgs. Would be interesting to hear if that mirrored your experience or not.

I'm not so convinced based on my experience. Seems like that could be the ideal, but in practice for such a large institution it seems difficult to implement. Sharing the word that this team over here is using XYZ and seeing good results to that team over there is challenging.


Ah okay, thank for the that :)

I sort-of assumed that for big purchases it would be a big procedure, as you've stated.

I had it in my mind, there'd be like a lower threshold (i.e. licenses for one or two things), which would go under the radar and just be purchased when someone requested it (or end of year "budget needs to be spend" - you know, that sort of old school "budget per department - spend it or you lose it" sort of thing.) :D

This also went with the idea that the "3 x CIQ Rocky Enterprise Linux Per Person Advanced - Annual Subscription Service Period" would be a small purchase - though of course, I don't _really_ have any idea what this means - could be a massive 24x7 support with 3 Rocky Linux support staff on-call - who knows :)

> That, and ROCKY can probably now have a NASA logo on their website under "customers", which adds a bunch of credibility to the project, honestly, whether it's necessarily deserved or not based on 3 desktops.

This on the other hand is verry interesting!! Thanks for pointing this out :)

Edit: sorry, I misread your comment :( Thought you said their logo _was_ on the Rocky site :P


Is this potentially the reason RH won't be providing public sources for RHEL any more? RH's bread and butter is USGOV. NASA using a direct competitor that can only exist by rebuilding RHEL sources, and this use becoming easier isn't great for them.


That seems very likely, either for personal preference or for testing the distribution


They didn't purchase licenses. They purchased support.


Yes for 3 workstation, they are probably testing the support


Or possibly a project installed Rocky Linux because it was easier, faster and cheaper than procuring RHEL licenses. It was found in an audit and now they had to procure and switch to RHEL or procure support for Rocky Linux. A complete wild ass guess but I have seen it happen at other agencies with open source software.

Since I retired on Friday, my experience with government IT and procurement is slightly out of date but based upon my knowledge of it until that point I would consider it a better wild ass guess than NASA procured 3 Rocky Linux workstation support contracts to test it before making some agency wide switch.


> installed Rocky Linux because it was easier, faster and cheaper than procuring RHEL licenses

Thanks, I almost spit my coffee.

Some years ago I read some docs on how exactly RHEL licensing works... guess the team which Oraclised MSFT in 2010 moved to RH.


The complications in getting a license that I was thinking of was those imposed by the government procurement process and not Redhat.


Yes, it's obvious, but in essence Rocky licensing is downloading an .iso and reading LICENSE.txt and comparinv this to anything RHEL related is what gave ne a chuckle


The pdf mentions renewal, so maybe not even that. Just 3 people renewing a support contract.

"The item described in the table below is for the FY23 CIQ Rocky Enterprise Linux Renewal."


Wonder if the timing of this is a coincidence, considering that RedHat just announced they're stopping releasing RHEL sources except to paying customers: https://www.redhat.com/en/blog/furthering-evolution-centos-s...

Link submitted by someone at https://news.ycombinator.com/item?id=36417070


I'd been wondering when that was going to happen. There's got to be a point when you grow tired of Oracle, Rocky and the like taking customers from you using your very own sources. That said, it didn't seem to be a major issue before the CentOS mess they created all by themselves.


It's important to be accurate when considering the actual situation. With that in mind, please elaborate on the phrase describing "sources" as though they are Red Hat's "very own". Is this honestly the case, or are you forgetting something truly relevant about the sources which Red Hat uses to build its products, and the legal obligations which apply to any users of those sources?


You have to pay RedHat to use its software. Only giving source code to paying customers doesn’t violate the GPL because of this. You only have to release sources to the users of your software.

RedHat has been giving them away to everyone; but they are not legally obligated to do so


As an interesting addition, Red Hat can't stop their customers from sharing GPL and LGPL package sources.


Correct.

To add to this, Red Hat does offers a free developer tier subscription, which allows individuals to download Enterprise Linux free of costs, but with strings attached. One of those strings, somewhere in the fine print is legal language about the restrictions of redistributing the software at the free tier. I'm not clear on the exact details, so I won't elaborate & please don't quote me, but it's my understanding that it's somehow a violation of terms redistribute. I'm not sure how that works with the GPL, and I'm guessing the bits ARE still shareable under the GPL, but the separate contract with Red Hat makes that a breach of contract. The mind boggles.


"Very own" as in you've put in the work to curate, test, and bundle together a ton of disparate software to create a cohesive final product. Let's not pretend that there isn't a ton of value in that which these other companies are taking advantage of.


But oracle or rocky can just subscribe to RHEL and continue having access to the source code, right?


I suspect that republishing the sources will be against the terms you agree to in your subscription.


I don't believe the terms of service restrict (or even could restrict, see GPL) the redistribution of _sources_. Compiled binaries are another matter.


The game that has been spelled out elsewhere is that the subscription can be terminated for this violation of the subscription terms of service (which is not the same thing as the software license). This termination would prevent you from receiving future versions of binaries or sources.

So, RedHat can satisfy the letter of the law regarding licensing, where source is distributed under GPL to those who received the binaries. With each incremental binary distribution they satisfy the GPL rules and makes source available to the recipient of binaries, but they may cut off a customer as soon as they detect that the customer is actually exercising their full GPL rights. So that customer loses access to future binaries and corresponding sources.

This attitude violates the experience many of us take for granted about free software and the GPL, where we expect a community to be engaging as an equivalence class of members sharing the same rights. But the GPL started with provisions enabling this narrower sense. The source distribution could be offered individually to each recipient of binaries _or_ made generally available to the public independent of any existing exchange of binaries between two parties. So the GPL allowed the recipient to transitively distribute the sources they have received, but does not actually obligate the original distributor to also provide sources to other unrelated parties.


> So the GPL [...] does not actually obligate the original distributor to also provide sources to other unrelated parties.

It seems important to note that the Debian Free Software Guidelines as applied by the Debian project specifically exclude any software that requires providing source to unrelated people this way (the “desert island” and “dissident” tests[1]). In practice, the “unrelated” people in license terms of that kind are usually the original copyright holders.

[1] https://people.debian.org/~bap/dfsg-faq.html


They can’t, I think, but nothing prevents your support contract (and thus access to subsequent updates in any form) from terminating when you release the source. You can do it, IBM will just refuse to do any more business with you from that point on.

This kind of GPL not-a-workaround always made me feel a bit queasy, but open access to RH code (even if a bit stale and hard as hell to get to build) alleviated that feeling somewhat. I guess it no longer does.


They can't add more terms on the GPL and call it GPLv3.


Yes they must abide by the terms of the licenses of the software they redistribute. But they could start licensing the parts they write more restrictively, like RPM spec files. I don't think this will happen given how much back and forth there is between RHEL and Fedora, but IBM is in charge now.


I believe the GPL - which redhat must agree to when redistributing binaries - prevents redhat from creating that sort of restriction.


Well, they killed CentOS so I would expect nothing less. What would the recourse of Rocky/Alma be then?


I wonder why they didn't follow Fermilab and CERN and choose AlmaLinux.

https://news.ycombinator.com/item?id=33904336


Because Alma has ties to Russia.


This is totally FUD


No, it's not. Alma was founded by CloudLinux, which is a Russian company with a Russian owner, mostly Russian and Eastern Europe developers and fake US address. As of today, CloudLinux still plays a critical role in Alma.


Aside from CloudLinux's involvement with AlmaLinux[0], I can't find much online that verifies any of this information.

For those of us interested in investigating, could you provide any information to support these claims? It looks like FUD when these claims have no backing information associated with them -- but if there is more information available I would love to know.

According to his Linkedin[1] and an interview[2] the CEO of CloudLinux -- Igor Seletskiy -- lives in California. From some brief reading it seems his company is remote and I think I saw some content that he may have Russian employees, but didn't see anything notable that was worth linking here.

[0] https://en.m.wikipedia.org/wiki/CloudLinux_OS

[1] https://www.linkedin.com/in/iseletsk

[2] https://lowendbox.com/blog/interview-with-igor-seletskiy-ceo...



Check the address listed as headquarters: it's a residential area https://www.cloudlinux.com/about-us/

There's hardly any job openings now but there used to be a lot that required or strongly recommended Russian language. A couple of years ago, I was on the verge of working for TuxCare.


No fake US address. He does live in Florida for decades. Know him personally from cPanel days.


Close to BVI


Wherever you're getting this information from is woefully misinformed to the point of just being plain wrong.

CloudLinux is a large financial sponsor of AlmaLinux still, and probably always will be. The rest of what you said couldn't be farther from the truth.


Are you willing to bet your house?


What I find interesting is all the bureaucracy needed to get 3 licenses for a "Free" RHEL Clone.

I wonder of Rocky got some real $ from NASA ?


They didn't get 3 licenses for a "Free" RHEL Clone. You can't purchase licenses for Rocky Linux. What they purchased was support.


What a weird way to spend federal money. I say it's weird because NASA almost certainly has its own internal Linux support paid through overhead in one form or another, and again the incremental cost of adding "developer" support from Red hat is way, way cheaper than Rocky.

Finally, I doubt Rocky's support can perform as well as Red Hat's. No, I'm not talking about the people who talk on the phone when you break something. If you find a bug in a package, will Rocky be able to quickly and effectively upstream the fix, or will Rocky end up maintaining you on a custom patchlevel forever?

I doubt Rocky has the ability to truly fulfill 24/7 support. It's difficult to build a deep bench for support, and nearly impossible to make sure you're keeping up 24/7 capability for other Maintenance Engineering type tasks.

And if none of these reasons are important enough to stop you from paying them, maybe the thing you're doing isn't important enough to warrant paying for support anyway.


Former NASA/JPL employee. I can say with gusto that Linux was never really supported by overhead. Projects pay directly for cloud services, servers, computer time, equipment for field testing, etc. Those are where rocky Linux would be used. At least in my experience, the usual burden funds are used for vanilla windows desktops or laptops for employees, which works fine for 90% of the org. The rest write proposals to get non standard equipment in one form or another. This drastically reduces overhead costs and allows results driven grants to buy only necessary equipment. Same as any other gov funded research-related capital purchase. At least in theory.


To be honest, you sound more like a Red Hat fanboi than someone who is making persuasive arguments.

For one, you're implying that Red Hat can fix bugs and can upstream them more quickly than Rocky. Have you seen the amount of time it takes Red Hat to move fixes in to sources? That alone is silly to even mention. I don't know how long Rocky would take, but in the entire world, hardly any companies take as long as Red Hat.

Two, do you think Red Hat "truly fulfill(s) 24/7 support"? Sure, you can get someone on the phone, but how many hours will it take before you can get someone who knows anything? All that navigation and escalation is time consuming. I'd rather wait for an expert who knows the area of my problem intimately to be woken up to contact me after an hour than be appeased by being on the phone for three hours explaining a problem to person after person who won't directly escalate and wants to hear the problem explained.

"maybe the thing you're doing isn't important enough to warrant paying for support anyway." Huh? That's just ridiculous.


To be honest, you sound more like recent college graduate with minimal (no) experience working in enterprise.

You probably don't realize this given your lack of experience, but when you are running FOSS powered/adjacent enterprise equipment (think storage systems and compute fabrics) the bugs you encounter are often not things that simply show up on the first page of google.

There often isn't anyone to "wait" for (other than the resources you pay for), which is why if you value uptime and availability, you PAY for a support contract.

I was a staff level systems engineer at a major movie studio in California (owned a 24+PB tape library and a few petabytes of fast storage), and I can confidently say all of our dedicated support assets were valuable members of our broader team (first name basis, ask questions/raise issues with a text message, etc...).

In the rare occasion an engineer wasn't cutting it we would have our TAM get us a new one. Typically you are literally paying these people's salary as part of your contract so these are decisions that you as a customer actually have a lot of input into.

If you'd rather wait for someone to answer you on StackOverflow that's a decision you can make, but if you pull that move in any serious enterprise you will terminated pretty quickly.

The whole point of Rocky is to be bug-for-bug compatible with RHEL; your opinion on who can fix bugs faster is completely orthogonal to this discussion.


I'm flattered that I sound like a recent college graduate :) I've been a Unix systems administrator for more than a quarter of a century.

Wild generalization: Red Hat is mostly bullshit. Their support is good at handling the myriad quirks and bull they've intentionally added to their OS to differentiate their product, but they're not so good at things beyond that, in my experience. I'm generalizing, of course, but in too many instances Red Hat has been unique in having issues that other common OSes don't. In many environments I've set up proper servers with other OSes for comparison so I'd know when something is a Red Hat-ism.

If you're somehow authoritative because you've worked for a major movie studio with petabytes of storage, then I suppose I'm authoritative, too. I have and still work for major movie studios and I handle petabytes of storage, too :) Perhaps we know each other.

Red Hat couldn't get their OS to see all the zones of a StorNext, while at the very same moment identical hardware next to that Red Hat system running SuSE worked fine. Drives were swapped between the systems, and the problem followed Red Hat. It was a clean install. Red Hat support told us they couldn't do anything and said it was a StorNext problem. We contacted StorNext, and they were even willing to work with Red Hat. Red Hat required us to have a three way phone call to include the StorNext people and wouldn't answer their emails. I had to create email accounts for the StorNext people. It was extremely unprofessional and nothing moved on this until the company threatened to cancel Red Hat support entirely.

Unfortunately, my other experiences were more reminiscent of dealing with AT&T than dealing with an organization that wants to make things work.


> Have you seen the amount of time it takes Red Hat to move fixes in to sources?

They will absolutely cut you a patch to use on your systems. Upstreaming does take time, yes, but that doesn't mean the person experiencing the problem has to wait to get their issue resolved.


> I say it's weird because NASA almost certainly has its own internal Linux support paid through overhead in one form or another

Why do you presume this?

(Former NASA contractor here: they don't.)


Not who you are replying to but I would expect any large organization (eg: almost every university and large research institute) to have a competent IT team that can help with most problems.

I would not expect them to make patches to packages or the kernel and I do not know how IT works, but I suspect this is the reason they presumed what they presumed.


Honestly I suspect rocky support to be faster than red hat.


I suspect Rocky support is three people in Europe somewhere with maybe one or two others in North America. I also suspect Rocky has no Support Engineering group, so their own product engineering resources must be redirected every time a customer needs a patch.

If Rocky does have a Support Engineering group, they aren't regular committers to the kernel, and so most common bugs found will take much longer to be upstreamed. This increases the load on Engineering because they'll have to carry that fix until it makes it into upstream and the customer upgrades.

I have seen no indication that Rocky has a robust QA infrastructure for customer patches.

Note the Rocky support model allows each "person" to only have two cases open at a time. This indicates they're forced to limit capacity.

Let's say I'm wrong about all of this. Let's say Rocky support is better than Red Hat. I've seen what happens when support scales from a small company to a large one. If ever you got good support, those days are over if Rocky sees success and the support group scales.


It takes a lot to design and build a car and manage its dealerships. It takes orders of magnitude less effort for a mechanic to maintain it, and even less to simply paint and clean it. Red Hat builds this car - Rocky repaints it and acts far more like Jiffy Lube than like a dealership. Have you ever seen how long it takes, and how much more it costs, to have your oil changed by the dealership? Those focusing on the specialty are able to deliver on it more efficiently and with less expense to all involved.


I've seen what happens when you take your car to Jiffy Lube and the minimum wage oil tech they hire doesn't tighten the oil drain plug all the way.


Or strips it.


The support is provided by a company called CIQ rather than the engineers behind Rocky itself. You can even see their support tiers on their website (NASA picked Advanced)

https://ciq.com/support/rocky-linux/


The difference is with RH you can often get escalated up to the engineer writing the thing. I know I did when working for companies paying for that support. There's usually a few layers of rather useless L1 and L2 though.


Those lower layers are a significant anti-feature for a customer needing quick, effective support, and it is only occasionally/rarely the case that a key author or maintainer is actually employed by Red Hat.

Most things delivered by Red Hat were not written by Red Hat and where they are, that delivery tends to be a rewritten, renamed project which went from community-driven (FOSS made to serve the user) into a corporate-driven model (gate-kept software which serves primarily as a profitable subscription-dependant product).

The trade-off is effectively for legal liability transference more than for genuine supportability. I would rather get eventual real support from the bazaar of people who wrote a thing (knowing, intimately and technically why) than liability-waving support (assuming successful escalation) from those who rewrote it in the cathedral with divergence intentionally created for corporate control and profit.


Actually most of the important bits things delivered by Red Hat are written by Red Hat employees. Or at least maintained.


What makes you think a company with 89 employees that just clones Red Hat's source code would be faster than Red Hat with 19,000?


Precisely because it has 89 employees instead of 19,000, it can be MUCH faster.

Have you ever tried to get support from Red Hat? Honestly? The first couple of tiers are often people who are barely computer literate. I've talked with several people who - I'm not kidding - had no idea what the Power architecture is and had no clue that there are architectures aside from x86.


Have you tried getting support from RedHat as an actual enterprise customer rather than an SMB?

Obviously if you pay for the lowest level of support, that's what you are going to get. Serious businesses pay for dedicated account resources. This typically includes a Technical Account Manager (TAM) and a dedicated support resource that works along side Professional Services to take care of your install.

When done _properly_ you will have nearly instant support from an excellent engineer.


Not sure who you were talking to but Red Hat support doesn't use tiers, cases go directly to SMEs on the feature and from there the only escalation point is the actual engineers working on the lines of code affected.


On multiple occasions, I've had Red Hat take 6 months to resolve a paid support case to fix something that already had a fix plus test cases committed upstream, and that could be cherry-picked into their version with no conflicts.


"Take 6 months" sounds a lot like "distribute the fix on the next minor update rather than doing so asynchronously".


Price's law, for one. Experience with the general effectiveness of large companies, especially their support teams, for another.


If Price's law is at play, then there will still be more competent people doing the work in Red Hat's support team than Rocky's.

How likely is it that there's someone on Rocky's staff of 80 who...

a. is an expert in the code, and has worked on it on customers behalf before

b. is able to effectively support a customer in addition to fixing a bug

c. is awake at the time you need them

d. isn't totally underwater with other issues

How likely is it that there's someone in Red Hat's staff of 20,000 (or whatever) who fit the same criteria?

You'll need experts in kernel, services, storage, networking, filesystems, virtualization / containers. Enterprise support shouldn't be done by generalists, it should be done by someone who has depth in the particular problem space.


Red Hat is supporting a lot more subscribers, I'd think? I have no experience with Red Hat phone support, their product has always been great and their online resources answered any question I had.

My impression from access.redhat.com is that much of the written material comes from support given to actual customers.


Red Hat (any many other companies) follows "knowledge centered support". [1] Essentially the knowledge base is the "workspace" or scratch paper of the support case, and as information comes in the case, the information should be organized in the knowledge base solution such that by the end, you've got the bones of a solid solution other customers can follow.

It's a whole Thing that People charge huge sums to implement, but the idea itself is pretty solid.

[1] https://www.serviceinnovation.org/kcs/


The mythical man month.

More people can very easily lead to nothing getting done.


This doesn't really apply to support. With support, there is generally 1 engineer assigned to one customer for the life of the case. If a patch is needed, a software maintenance engineer may be added. You can't throw extra people at a support case and expect it to complete faster. You can, however, throw more people at a support TEAM and handle more cases.


So Rocky needs to assign a single engineer to NASA support then. How does 19,000 people help with that?


That's typically what RedHat does...

If you're a big account you get your own Technical Account Manager (TAM) and a dedicated support engineer.

This is like bread and butter for anyone with any experience in enterprise.

How is HN so green in this regard?


No I'm saying if you're assigning *a person to a contract* Rocky can easily do that too.


Because surely they have other customers than these three workstations.


It's not 3 workstations, it's 3x per person support.


Point still stands, when you have an 80 man team you can’t have many customers who require your undivided attention.


Did you watch the movie 300? Then you know


> No, I'm not talking about the people who talk on the phone when you break something. If you find a bug in a package, will Rocky be able to quickly and effectively upstream the fix, or will Rocky end up maintaining you on a custom patchlevel forever?

I'm intrigued. What makes you think Rocky will be faster than Red Hat? How does Rocky handle that situation? Do they have COPR repos or similar that you add to your system? What do they do if the patch gets rejected upstream?


Is your car dealership faster at changing the tires and oil on your car than a tire or oil-changing company would be?

RHEL is effectively a bunch of FOSS bundled and rebranded. Rocky is effectively RHEL with another paint job. If a patch gets rejected upstream, Red Hat is known to reject the author/maintainer's rationale for this rejection and, over years, may then extend, embrace, and extinguish that community, replacing and repainting it as their product.

Sometimes this is a community-serving change (i.e., docker -> (OCI+) -> podman), and sometimes it's a community-replacing change (i.e., Kubernetes -> Origin/OKD -> Openshift). In all cases, it's a redeclaration of who actually is the most knowledgeable expert. Management re-assigning experts is not necessarily as aligned as the merit of an original engineering team/community providing their solution.

I'd rather Rocky's solution than the walled-garden in a cathedral courtyard approach. If something cannot be pushed upstream (i.e., user-hostile defaults, vendor lock-in, or arbitrary rebranding), this blockage is sometimes a feature we all want for FOSS, even while it may not give privileges to those who can throw more money at the problem under the condition it will then be able to extract more profit.


The purpose of Rocky is to be bug-for-bug compatible with RHEL. What makes you think they would ever attempt to upstream something?


We do all the time.


They probably do it like Debian. "Here is a patch, looks good, just ship it."

With Red Hat it would have to get approved and tested first.


Out of curiosity what do others see as viable Red Hat Linux alternatives?

If I was choosing a server OS today, I personally would probably pick Debian.


There isn't really any, only Debian/Ubuntu gets close.

RHEL and its clones are supported with updates for 10 years, often ship with "more modern" server software than Debian does (e.g. firewalld, podman, systemd etc) and DNF, whilst slower, still blows apt in terms of feature set (e.g. transcriptional, roll-backs, history, offline upgrades)

There's also SUSE.


RHEL saying '8' or '9' is supported for 10 years is misleading af. Each 'minor version' (which are, in fact, sometimes large changes!) is supported for it's own specific time period. Odd numbered (8.5, 8.7) for example are only supported for 6 months. https://access.redhat.com/sites/default/files/images/337_rhe...

It's terrifying actually.


Yep, the "10 year support" for 8.8 means that you'll upgrade to 8.10 (to-be-released) and then that will be supported for the rest of the duration.


> often ship with "more modern" server software than Debian does

That's not really true. They often ship with Red Hat specific software (like Podman is) replacing widely used software (e.g. Docker, to which Podman still isn't feature complete even though it has been shoved in everyone's throats for a couple of years now), but "more modern" will vary wildly based on the release date.


> RHEL and its clones

My understanding is that the problem with RHEL clones is just that, that they are clones.

From what I've read, the problem is that RHEL will often make changes to suit their various enterprise customers, for example I seem to recall reading somewhere that OpenSSL was one of many things "touched" in that way.

I hear many good things about DNF vs apt, but I'm too worried by the skeletons in the clone cupboard.


> There isn't really any, only Debian/Ubuntu gets close.

> There's also SUSE.

So, why doesn't SUSE get close?


The Marketing Gap. Also the unfortunate detour with Novell and MicroFocus for a few years...


Give them a few more. They acquired Rancher, k3s, have MicroOS, and ALP in the pipeline. I'm rooting for them.


There are a number of shops where it is advantageous to have a paid RHEL subscription, mostly when there are non-technically-focused management involved. When there is an issue I've had managers in the past say "Quit dicking around on the internet and get someone on the phone".

For these shops, you don't need the paid enterprise offering across the board (dev, testing, etc) so that is where an EL rebuild is handy. In those cases you don't want too much different between prod and test/dev, but you don't need the paid support or priority patches on internal test hosts.


Do the managers think the person getting the ticket at RH will not be dicking around on the internet?


Red Hat employee here. They most certainly are not. The people on the phone have been through a lot of training, they are all whizzes on the command line, we have an enormous in-house knowledge base that's updated continuously, and there's a clear escalation chain as well (as an engineer, I get customer cases that resist known resolutions, so I will be the one "on the internet", searching for known MariaDB issues, things like that).

RH would not be of much value if the support staff were not any more effective than the average end user.


That's my point, the guy who had the issue in the first place is doing the same thing you are.

It's only the boss that wants to pay for a safety net, forcing the issue of raising a ticket is their validation of money well spent I highly suspect. We're all human beings who are quite technical on this forum, you work for RH, the other guy works for someone else.

I have a suspicion that RH support is what the boss pays for as a backup should their employees be unable to solve a problem or decide to go to pastures new, as people do from time to time.


> That's my point, the guy who had the issue in the first place is doing the same thing you are.

they're not, because they have no idea what they're looking for, they lack the subject knowledge and expertise, and they werent involved with directly producing a lot of the open source components they are having problems with.

By "engineer" I mean, "we are the people that wrote the actual open source components they are having a problem with". It would not be in their interests to hire us directly because 95% of the time the regular support people can do everything they need without things escalated to engineering.


I very much doubt you have all the authors, some components are RH creations, but not all. In any case, a company hiring one or two open source contributors covers a lot of gaps as they're the type of people who can do deep investigations. For systemd, for sure, there's inhouse development there.

My gripe about the original thread was bosses interrupting that investigation and telling the onsite staff to open a case elsewhere, IMO, that's disruptive behaviour.


> My gripe about the original thread was bosses interrupting that investigation and telling the onsite staff to open a case elsewhere, IMO, that's disruptive behaviour.

I dont think this really happens, when we get these customer issues and the people who have tried to fix the problem are on the phone, they are lost / panicked / at a dead end. They are not like, "yeah we think it's ABC but our boss told us to call you". that's not really a thing. if customers can fix the problem, they do. they are not in a hurry to call redhat for things they can do themselves and they're not really supposed to since their support minutes are a finite resource.


Nobody would do that when logging a ticket because it shows internal company disagreement which often doesn't show the company in a great light. Nobody does that.

What typically happens in this situation is the chap logging the ticket gets pissed off and says to the boss, "ticket logged it's with $vendor, I guess it doesn't need babysitting so I'm working on other tickets now".

The boss in this case is happy as it shows their earlier investment was a fine idea. The employee though is disgruntled as their toys were taken away and things it's not such a fine place to work.


I'm kind of shocked the average HN user seems to be so incredibly ignorant on this matter.

Is this an ego thing?


Generally no, based on past experience with RH support, they're not just dicking around on the internet to find the answer. Most of the folks I know at RH support have been there 5+ years, and are life-long users and community participants in the project they're supporting.


I once worked in a large open-air office that had on-site support from VMWare (this was over ten years ago). The VMWare guy was a doofus and some of the 20 something year old IT staff appeared to know more about his product than he did. Provided no-value except for "knowing people"...One day he wasn't there and I asked if they finally fired this useless bozo and I was informed that he went to work with Redhat-support and that a new VMWare guy was inbound. Shook my faith in RHEL for years. Honestly I'm still not sure I trust Redhat after they hired such a loathsome fraud.


My RH support experience was bad enough for me to switch to Rocky Linux. I suppose people's experiences in life can be different?


Managers think the blame parcel has been moved to a supplier


It’s called “risk management” in corporate speak.


Risk management is nothing to do with removing the risk from the company, indeed increasing risk is often acceptable. It's about removing risk from your department.


Yeah, this does seem to be the prevailing way in management layers.


I once had a vendor support tech tell us to add build configuration variables like ARCH and CROSS_COMPILE to our .bashrc. This is a terrible idea for a variety of reasons I won’t get in to.


There is definitely a value if you don't have a bunch of Linux nerds on board or desire to be the one that tracks why some random bug happens.

For just ease of management and customization can't beat vanilla Debian


last I knew, Debian requires config setup, not optional, not guided. There is no "defaults that work" for a network server. Is this still true in 2023 ?


Your question seems to be presupposing a bunch of things.

If you run the standard installer from a vanilla USB image, you will be asked questions about the config, most of which produce reasonable results if you just hit <enter>. Some of them will fail in certain environments. (No DHCP server? You will need to enter IP networking details.)

If you want to install a fleet of Debian machines (or VM images), you set up a PXE server and a set of answers to the installation questions, or you build an image for direct deployment, or you use the Debian VM image, or... whatever: there are a lot of possibilities that will work.

Most of this has been true for at least 3 Stables.


For business? https://www.suse.com/solutions/business-critical-linux/

For home server use: Tumbleweed, Leap or MicroOS depending on your personality. :)


Debian is relatively problem-free compared to RHEL; and update-in-place also works.

Sooo much less fuckery, packages are just there and don't require installing/buying some random addon just to have DRBD+pacemaker cluster (that RHEL for some reason put into separate repo and subscription).


I went with Ubuntu because the long-term support Debian at the time would have required me to hand-compile or add package sources for the version of whatever it was (php?) I needed.

Ubuntu LTS is long enough for me, and seems to work without complaints.

At home I run Gentoo because I like to fomit and funroll.


Go with an immutable distribution, something like CoreOS. Then run everything in containers.

Major version upgrades have less risk of breaking, while rollbacks become an option. And the attack surface is lower.


Attack surface will be sum of CoreOS surface and surface of containers. It can be lower only if surface of containers will be negative. :-/


Professionally, I'm running everything on Debian because it's always been around, always rock solid, isn't going to make any stupid decisions for profit, and containerizes well. If I'm at a place that's willing to pay for RHEL I will go RHEL, because the support doesn't hurt and they do a great job contributing to upstream projects.

For personal machines, I don't mind Fedora Server (or CentOS Stream) because I can handle the more frequent upgrade cycle.


Devuan and Void Linux are systemd-free alternatives. The first one is a Debian fork, so should be very familiar to anyone with Debian/Ubuntu experience.


AlmaLinux is being heavily floated as an option in my place of work. We have RHEL7 and 8 licenses but due to pricing might switch.


If stable then Alma or Rocky.

If unstable then Centos or Fedora.

I also have to use Ubuntu and Debian, and they are miles below Redhat derivates.


Fedora, especially one of the recent immutable versions.


popOS - it really surprised me after long time Ubuntu, before Debian user.


popOS as a server operating system? That doesn't make much sense.


My bad, somehow I missed server part.


How do I choose between Rocky and Almalinux? Or is it just not a big enough difference to worry about?


They have different organisation structures, where Alma is a 501c Non-profit and Rocky is a backed by a B-Corp, so in my head I read one of those as a community linux ie. Debian-feeling, and the other aims to make money ie: Ubuntu-feeling. That's an extremely crude interpretation though, both need cash to support themselves, so both need to make money some how.

Day to day they should be by design, interchangeable. Rocky has a nicer logo... This kind of ugly exchange on reddit[0] always sort of soured my impression of Rocky though, didn't really pass the lebowski test.

0. https://www.reddit.com/r/linux/comments/qv6mg2/were_the_alma...


Alma has been faster with releasing updates and security patches from what I’ve seen.


This seems to be true. However, looking at https://rockylinux.org/news/rocky-linux-9-2-ga-release:

9.2 Release for PowerPC (ppc64le) architecture held back

During testing, we discovered an architecture-specific issue on ppc64le systems with the bundled version of Python 3.9. This issue not only prevents installing, but may break existing installations.

This issue is reproducible on CentOS Stream 9 and RHEL 9.2. We have opened a bug report upstream at rhbz#2203919 and are working to fix the issue.

Seems like they hold back releases to do additional testing. In this case, they avoided a bug that was present on RHEL (and presumably AlmaLinux).



regardless of the status of the bugzilla. That _is_ a bug and a regression in RHEL 9.2.


Last I looked, there was no real difference. I don't know if there would be any politics preventing it, but I would not be too surprised if they merged at some point.


Almalinux releases faster, and patches faster than Rocky.


Out of curiosity, what makes this notable?


It's another link in the Red-Hat-CentOS-drama chain.

I think many IT employees have a vested interest in Red Hat and the direction of "enterprise Linux". Seeing an organization like NASA show a willingness to use a Red Hat alternative gives credibility to that alternative.


Giving users the ability to have a choice in the direction of "enterprise linux" is a large part of the reason that CentOS Stream exists. Previously if you wanted some bug fixed or some feature added you needed to beg Red Hat for it or hope that a paying customer needed it too. Now external parties can contribute to the future release of RHEL (and Rocky, Alma, etc.)

So let's say that some Rocky or Alma Linux user or developer wants to make a change. They are RHEL clones, so they can't do that. But they can now contribute the change to CentOS Stream, which is the upstream for both RHEL and their own distro. That change can be reviewed and tested by the community + Red Hat, and then land in RHEL a few months later.

This isn't theoretical, to use one example, Facebook uses CentOS Stream in production and contributes back a significant amount.

Disclosure: I do work for Red Hat. But it never made sense to me why people say that the "CentOS community was destroyed" when it didn't really have much of a community, it had _users_, which is different. It has more of a "community" now than it did previously, because the development is actually open as opposed to Android-style "throwing bundles of code over the wall".


> Giving users the ability to have a choice in the direction of "enterprise linux" is the entire reason CentOS Stream exists.

Pretty sure it's just for free beta testing


More than one thing can be true at a time. GP's explanation is one of the reasons. IMHO it's not the primary reason, but it is a legit entry in the "pro" column


Of course, if we do get an established leader, it will ultimately see the same fate as Centos.


It's by the guy who setup CentOS originally, no?


I think this is notable because in the wake of RedHat changing CentOS model changing from stable to bleeding-edge, there has been a niche to fill in the enterprise-grade Linux space. Esp for people running on-prem infra e.g. HPC. Do we settle on Rocky? Alma? Amazon Linux2?

NASA seems to be taking the lead here and saying "we're going with Rocky". I think this move will encourage others on the fence to do the same, and build more of a community around the product not owned/maintained by Amazon.


>bleeding-edge

CentOS Stream is not "bleeding edge". Every update that goes into CentOS Stream has already gone through the Red Hat QA process, and the updates are restricted to only the kinds of things which previously would have gone into RHEL. That is, mostly security updates and bugfixes.

It's a little bit ahead of RHEL, but it's not like Fedora, much less Rawhide.


> in the wake of RedHat changing CentOS model changing from stable to bleeding-edge

CentOS is now bleeding edge? Would you describe Debian Stable as bleeding edge also? On your contiunuum, where would you place Arch and Fedora?


> Esp for people running on-prem infra e.g. HPC. Do we settle on Rocky? Alma? Amazon Linux2?

Given the stage of the lifecycle, you'd be pretty silly to adopt AL2 on-prem at this point. AL2023 probably won't be appropriate either. Rocky is the way to go.


If Red Hat hadn't sunset CentOS/RHEL parity, none of this would have happened, maybe. Rocky wouldn't exist.


Would this have happened if IBM hadn't bought Redhat?


Probably. Red Hat acquiring and then maintaining a failing CentOS was one of those solved a problem at a couple times things that just didn't make a lot of sense in the long run (and was almost certainly a mistake).


CentOS was great and I'm still using CentOS 7 until EOL. I learned a lot from "the mouse" in the CentOS support forum, and I hope the CentOS people made a living from the deal. I've got no bad feelings towards Red Hat, as long as they weren't coercing anyone against their will. For new systems I use both Red Hat and Rocky.


Wikipedia:

> Rocky Linux is a Linux distribution developed by Rocky Enterprise Software Foundation, which is a privately owned benefit corporation that describes itself as a "self-imposed not-for-profit".

Is this only a structure set up to fund whatever needed to be done to rebrand RHEL? Or does anyone have shares that could be worth significant money?


The more I think about the RedHat source code restrictions, the more I think this could be the reason. RedHat has mostly tolerated the free rebuilds, but now that one of them is encroaching into providing paid support (RedHat’s bread and butter), that’s a step too far and it could have poisoned the well for everyone.


I don't understand why they don't buy RHEL. Rocky is essentially RHEL. If RHEL isn't supported there is no Rocky.


There's also the question: If I buy support from Rocky Linux and they provide a fix for something, then they'd need to upstream it, or how much can they allow Rocky to drift from RHEL, and for how long?

What if they surplant RedHat, can they afford to undermine RedHat long term, or is there a point where they fork?

Rocky Linux, and CentOS before it, is a strange product for people who want the benefits of RHEL, but don't want to pay for RedHat work. If you don't want support, then I think is fine, that the cost RedHat pays to be an open source company, but when you then turn to another company who rebadge RHEL and charge for support, it gets a little weird.


Is this the software equivalent of buying a generic drug? It's the same basic product but the details may make it work just as well or less well for any specific user. If it works for NASA and they save money, it seems like a win.


From the vendor side, it's more like the software equivalent of dropshipping. Slap your own label on something you didn't make and call it a day. If the original "manufacturer" goes under, you either pivot or fail too.

From the consumer side, if you buy a dropshipped product you don't get the benefits of working with the manufacturer, and are risking worse customer service.


No quite, the generic drug makers can produce the product from scratch without the help of original manufacturer. Also RHEL is contentiously being developed, once a drug has been designed, there's no requirement to keep it updated (ruffly speaking). Rocky Linux is dependent on RedHat staying in business.

The problem will only manifest itself if RedHats business is sufficiently underminded. Losing a few customers isn't an issue, and many of the users of Rocky Linux probably wasn't going to buy a RHEL licens anyway.


It's probably an evaluation in preparation for removing CentOS in ways other than buying RHEL.

Federal agencies can't really use CentOS after 2024-06-30, so everyone dealing with federal clients is working on an upgrade path (CentOS was a major solution due to piggy backing on RHEL certifications )


There is no Rocky Linux 7, and CentOS 8 is already EOL, so I think it's a valid question.


There's a lot of CentOS 7 in various USGOV and adjacent (suppliers/service providers) who are stuck facing purchase of RHEL8/9 licenses or migrating to other distros - as CentOS 7 going EOL on 2024-06-30 means you can't use it for federal systems without each federal client getting a waiver.

You have to either go with another distro or provide your own support (essentially your own fork).

Those who want to stay with RHEL-like but not deal with actual RHEL licensing are often going with Alma or Rocky.


Again, like large corporations, this doesn’t mean much. NASA is big and diverse, and each facility is unique and mostly independent, with lots of unrelated projects within a facility.


This headline reads backwards to me. While it would be true either way, it seems more intuitive to say "Rocky Enterprise Linux secures contract with NASA"


How new is sam.gov?


Very old.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: