The members of the kernel security team are not allowed to tell their employers anything that happens on the security list. They are there as individual members, not as employees.
And try to define "major distros" in a way that actually means anything viable.
If you just want to count users, then that would only be Android (everything else is a rounding error.) After Android, that would be Yocto, and then Debian. All distros after that are mere fractions of overall users compared to those 3 by number of running systems alone.
If you want to count it as "$ spent on Linux" then that cuts out Android and Yocto and Debian as those distros are free, and would focus purely on the tiny installed base of paid Linux systems, and cut everyone else out.
So what is a fair way to do this other than "we notify no one, and tell everyone to always update their systems to the latest stable releases that we support."
Especially as there is no way for us to determine your use case (i.e. if a specific bug is a vulnerability for you or not.)
Thanks for the reply (and thanks for the work you do)! Fair enough. And the issue is also that without some form of vetting you run the risk of disclosing the 0 day too early?
About that "That's the only policy by which all the legal/governmental agencies have agreed to allow us to operate in, so we are stuck with it.", you mean that if you disclose selectively, then you become liable for damages? or was it a more direct conversation with legal/governmental agencies?
And for a bug like this, what is the policy with backporting patches to lts branches? Since it was corrected in mainline on april 1st but only backported after the public disclosure. Do you delay backporting to minimise any attention on the security issue?
I guess that having a patch for that land on all the LTS branch would signal to any would be attacker that it's a significant security issue...
Sorry for all the questions but I'm genuinely interested.
If you want to talk about possible exploiting being done. Then Android is out (userland is crippled) and I guess yocto as well (same issue). Not that they can’t be attacked, but because mostly what is there is static. As it’s a privilege escalation attack, that leaves us with anything that is running code by unverified users (vulnerable server software, linux shell services, untrusted software you think you’ve sandboxed with user account,…). That put Debian, Ubuntu, Rhel, Fedora, Arch,… installation as the juicest targets.
Oh... thank you for the reminder to try running the C version of this exploit on an Android phone over adb. The curiosity is now killing me.
Edit: for context, I work in embedded and the aarch64 version (PR #42 in the repo) has successfully popped every device I've tried it against except one where I have a custom kernel to work around a driver issue and (looking back at my git logs) accidentally forgot to enable the user-mode API for alg_aead specifically. Lucky mistake.
They hide structures very easily, allowing programmers to accidentally put them on the stack or use them as parameters in functions where they shouldn't be doing so. By forcing "struct" on the name, it makes it more obvious as to what you are doing.
Did anyone think to actually ask the developer who is maintaining the LTS kernel versions why he made that change (back in February?), i.e. me?
{sigh}
No, I guess that would take too much effort, and wouldn't result in such a click-bait headline "LTS kernels are no longer supported for 6 years because it turns out no one used them." doesn't have that same fun sound...
> LTS kernels are no longer supported for 6 years because it turns out no one used them.
Looking quickly at Debian, old old stable (10) uses 4.19, old stable (11) 5.10, and stable (12) 6.1. All these versions are LTS kernels.
I don't think Debian uses LTS kernel branches as is, instead cherry picking patches, but it seems they do leverage LTS branches, at least to ease the back-porting I presume.
I'm probably missing something, and looking through the Debian mailing lists, I've not seen any discussion about the end of long term LTS kernels, but it seemed they did rely on it to a degree.
Maybe it was more of a nice to have, but from the outside (I'm neither a kernel or Debian maintainer), it leaves me a bit confused.
The one does not discount the other. In other words: disregarding the lts misunderstanding it still appears true that the Linux foundation is abandoning Linux kernel development. At least financially. Worth a look I guess.
Also: thank you for all your contributions to Linux for all these years!
The amount of resources and other stuff that the LF provides to the Linux kernel community has increased over the years, including last year. Just because new people are brought in with new projects (that the LF member companies want to host) does not mean that somehow less is being given to the kernel community at all. It is not a zero-sum game here at all, that's not how the LF works in any way.
Again, this would have been easy to verify if someone just asked us.
So to repeat, no "abandonment" is happening here at all, the opposite is happening, just like it has for the entirety of the LF's existence, support has grown every year.
Can you explain what this means? E.g. Debian very much sticks to LTS kernel versions in their stable releases. Were they not working with upstream, you? If they'd be doing their own maintenance, there's no need for them to stick to LTS versions, but clearly they didn't tell you. Any idea what's going on there?
> LTS kernels are no longer supported for 6 years because it turns out no one used them
I assumed companies use a lot of LTS software for servers to avoid frequent upgrades that always carry some chance of breaking stuff. Could you please elaborate more or link me to some place where I can read more about this?
No, the only rust code accepted into any released kernels is basic framework infrastructure so that someday, maybe, in the future, real functionality could be written in rust.
There are many out-of-tree examples of rust kernel code, but as of right now, none have been merged.
This already happens today at many companies. I write a few of these a year for companies that I do not work at, and have been for the past decade or so. It's not unusual and the companies that recognize that having their developers be a valued part of the overall kernel community is a very good thing to support.
Interesting. In the QEMU community I've never been asked for this kind of perf-feedback (with one exception in the last decade, which was related to a promotion case), and I don't think my fellow-developers would appreciate it if I passed on the request-for-feedback stuff my own employer's annual-review process includes. But if you're all happy to fill in those forms then I guess it works for you...
What is probably meant is "find easier to deal with" (hence the quotes around "manageable"). I'm only guessing, though - but in any event, let's not forget that not everyone around here is a native speaker of English! I can relate to the fact that it's sometimes difficult to express exactly what you have in mind.
I've described how we (the kernel security team) handles this type of things many times, and even summarized it in the past here:
http://www.kroah.com/log/blog/2018/02/05/linux-kernel-releas...
Scroll down to the section entitled "Security" for the details.
If you wish to disagree with how we handle all of this, wonderful, we will be glad to discuss it on the mailing lists. Just don't try to rehash all the same old arguments again, as that's not going to work at all.
Also, this was fixed in a public kernel last week, what prevented you from updating your kernel already? Did you need more time to test the last release?
Edit: It was fixed in a public release 12 days ago.
I don't know what you don't understand: EVERY single kernel fixes a few vulnerabilities. If you lazily refuse to update because none of those say "hint: there is a vulnerability here", then you are taking the deliberate action of skipping some security fixes. Greg's announces always say "all users must upgrade". If there was sometimes a different signal such as "all users must really really really upgrade", then for sure you would simply skip all other ones, as it already seems like you're waiting for a lot of noise before deciding to apply due fixes, and you would remain vulnerable to plenty of other vulns for much longer.
Here the goal was to make sure that all those who correctly do their job are fixed in time. And they were. Those who blatantly ignore fixes... there's nothing that can be done for them.
I obviously am very comfortable disagreeing with people who work on the kernel or adjacent software. Working in those areas does not at all make them correct, or even informed, especially with regards to security.
It's been working semi-well, and gives us a way to deal with longer embargo times (like months instead of weeks and days), but it does not integrate well into the linux-distro-like way of working just yet, which is an issue that hopefully will be resolved sometime in the future if the linux-distro members wish it to be.
> When doing kernel releases, the Linux kernel community almost never declares specific changes as “security fixes”. This is due to the basic problem of the difficulty in determining if a bugfix is a security fix or not at the time of creation. Also, many bugfixes are only determined to be security related after much time has passed, so to keep users from getting a false sense of security by not taking patches, the kernel community strongly recommends always taking all bugfixes that are released.
> Linus summarized the reasoning behind this behavior in an email to the Linux Kernel mailing list in 2008 ...
Since severity can be a moving target, it seems like there is no straightforward solution. With that said, by hiding the known ones, older distros don't have much of a hope in hell of getting all reported CVE fixes back-ported.
Why isn't there a public index mapping known CVE fixes to git commit IDs? This seems totally doable and would make the world a more secure place overall.
> older distros don't have much of a hope in hell of getting all reported CVE fixes back-ported
Older distros have always had a ton of privilege escalation bugs and I don’t think that’s ever gonna change. If you can’t keep everything updated, your machines have to be single-tenant.
While I did drop by a lot, you aren't describing me, but rather another Linux kernel developer who worked at IBM in our group at the time who is by far more brilliant and smarter than I.
I've never smoked, and at that point in time, I did not have long hair anymore (cut it off when I turned 30 which was before OSDL was ever started.)
The person who started udev (i.e. me), and the person who did the majority of the work on udev to make it into the proper solution for everyone (i.e. Kay Sievers), both agreed that it made more sense to move it into the systemd codebase in order for lots of duplicated code and functionality to be removed.
By doing this, we have made the maintenance and support for this core userspace tool much easier for everyone who was involved in working on it.
If the developers involved in a project do not do what you want with that project, feel free to fork the project as that's the beauty of open source! And hey, that's exactly what the eudev developers did, go use their fork if you want to, no one is forcing you to use the version from systemd, just like no one is forcing you to use any other open source program. It's your choice to do so or not :)
Why did you chose udev using libsystemd and not systemd using libudev (or made a different shared library, I'm not familiar with what happened)? Seems like this would have been a more widely accepted change and achieved the same thing.
As I understand there was some shared code between the two projects and udev was refactored into using systemd's code. Why not make systemd re-use udev code instead? Aka either expand udev's library or make a library out of the shared code that's used by both systemd and udev.
That way udev stays a separate project and you don't have to bring in a big library to use it. Systemd needs udev both ways so it changes nothing for it apart perhaps a bit of maintenance for a separate library for the shared code.
Expanding udev's library would probably end up a decent size library of other functionality, I don't think there is any good way around needing this. This is not code that could be deleted from systemd if it was separated. The way eudev has handled it is by doing what you describe and copying those shared functions out from systemd: https://github.com/gentoo/eudev/tree/34b2037d379e33f1cf79a34...
The result of that being there are two copies of the same shared code floating around, which is what they were trying to avoid.
>but if you check the code for udevd at this moment you can see a lot of use of utility headers that are not really appropriate for a libudev:
That's fair.
>This is not code that could be deleted from systemd if it was separated.
Why not? You could make a library out of those utility functions and use it from both systemd and udev (without having to include most of libsystemd which doesn't seem useful for udev).
I don't have a quote handy, but systemd maintainers have previously said they are not interested in also maintaining a giant public API of utility functions, on top of everything else.
Merge the portions that you know are correct and will have no affect on anyone else now, which makes future work easier as you do not have to keep those "working" commits up to date.
We do this all the time with kernel development, and is one reason why breaking changes up into tiny pieces is so powerful. We can take the pieces that make sense now, and allow the developer to redo the portions that are not ready yet, instead of having to reject the whole thing if it were done in one single "chunk."
Also note that the TTY/serial portions of this hardware support was already merged through the serial tree because they were independent and didn't affect anyone else.
The big "downside" is that it takes more work on the patch submitter side. But the benefits in the end are almost always more than worth it (easier reviewer time, easier time to track down problems, better development cycle as feedback can be more specific, easier evolution of changes, etc.)
I wrote a whole chapter in the book "Beautiful Code" about how this development model can help create an end result that is almost always better than the initial "huge" submission model. Check it out if you are interested, it should be free online somewhere...
My instinct would have been that it's easier for the submitter (as they have less to polish and test) and more irksome for the reviewer as they have to go through multiple rounds of submissions, but naturally I'll take your word for it!
This kind of discussion is always of interest to me, I'll check out the book, thank you.
Reviewing three changelists, which individually do only a single thing each, is in my experience much easier than reviewing a single changelist bundling the changes from all three.
This is true even if the same lines are changed multiple times. It's something you'll learn with experience, but it's also not even close. Break your patches up as much as possible, and everyone will be happier.
Not GP, but... In the context of recent event, [0] where not reviewing thoroughly enough some tiny patches had a major come-back-to-bite fallout, I can't help but wonder:
How, exactly are you expecting an increase in average patch size to help?
I did read through this debacle when it came out actually, I'm thoroughly on team Greg. I suppose my question was separate from malicious patches - I was interested in knowing if this incremental "merge tiny patches as and when they're ready" mode of development has ever caused issues with half-baked solutions affecting other parts of the kernel where perhaps it wouldn't have otherwise done so if the release was given more time for polishing and testing.
And try to define "major distros" in a way that actually means anything viable.
If you just want to count users, then that would only be Android (everything else is a rounding error.) After Android, that would be Yocto, and then Debian. All distros after that are mere fractions of overall users compared to those 3 by number of running systems alone.
If you want to count it as "$ spent on Linux" then that cuts out Android and Yocto and Debian as those distros are free, and would focus purely on the tiny installed base of paid Linux systems, and cut everyone else out.
So what is a fair way to do this other than "we notify no one, and tell everyone to always update their systems to the latest stable releases that we support."
Especially as there is no way for us to determine your use case (i.e. if a specific bug is a vulnerability for you or not.)