64 points | by zdw3 days ago
For context: I've been around in the Debian Rust team since 2018, but I'm also a very active package maintainer in both Arch Linux and Alpine.
Rust packaging is absolutely trivial with both Arch Linux and Alpine. For Debian specifically there's the policy of "all build inputs need to be present in the Debian archive", which means the source code needs to be spoon-fed from crates.io into the Debian archive.
This is not a problem in itself, and cargo is actually incredibly helpful when building an operating system, since things are very streamlined and machine-readable instead of everybody handrolling their own build systems with Makefiles. Debian explicitly has cargo-based tooling to create source packages. The only manual step is often annotating copyright attributions, since this can not be sufficiently done automatically.
The much bigger hurdle is the bureaucratic overhead. The librust-*-dev namespace is for the most part very well defined, but adding a new crate still requires an explicit approval process, even when uploads are sponsored by seasoned Debian Developers. There was a request for auto-approval for this namespace, like there is for llvm-* or linux-image-*, but back then (many years ago) this was declined.
With this auto-approval rule in place it would also be easier to have (temporarily) multiple versions of a crate in Debian, to make library upgrades easier. This needs to be done sparsely however, since it takes up space in Packages.xz which is also downloaded by all users with every `apt update`. There's currently no way to make a package available only for build servers (and people who want to be one), but this concept has been discussed on mailing lists for this exact reason.
This is all very specific to Debian however, I'm surprised you're blaming Rust developers for this.
What’s the additional security benefit of the explicit approval? A major problem with Rust (for me) is the multitude of dependencies required even for trivial software, which complicates supply-chain monitoring.
Auto-approval for crates gives me a bad feeling.
Guix is also a distro that allows for any number of versions of the same package globally, something that language specific dependancy managers do not.
Distors are there for a reason, and anyone who doesn't understand that reason is just another contributor to the ongoing collapse of the tower of abstractions we've built.
In Debian stable ripgrep on amd64 is currently on version 13.0.0-4+b2.
The relevant buildinfo file can be found here:
https://buildinfos.debian.net/buildinfo-pool/r/rust-ripgrep/...
It encodes the entire Rust dependency graph that was used for this binary, with exact versions of each crate.
for me: make an os out of the kernel
...they fail to integrate with dependencies written in any other language.
It's fine if you just want to sit a monoculture language software stack on top of a multilingual base platform. You can't make a functional system with one language alone, yet those who criticise distribution packaging architecture do so while simultaneously depending on this ability that language-specific package managers do not have. There is no viable alternative today. Most critics think they understand the general problem but only have narrow practical experience, so end up believing that their solution is superior while not considering the general multilingual software supply problem.
Nix isn't a solution either, because in the general case Nix isn't security-supporting arbitrary and multiple dependency versions either.
Distros (and the people who run most scales of IT org) want to be able to deploy and verify that the fix is in place - and its a huge advantage if it's a linked library that you can just deploy an upgrade for.
But if it's tons and tons of monolithic binaries, then the problem goes viral - every single one has to be recompiled, redeployed etc. And frequently at the cost of "are you only compatible with this specific revision, or was it just really easy to put that in?"
It's worth noting that docker and friends also while still suffering from this problem, don't quite suffer from it in the same way - they're shipping entire dynamically linked environments, so while not as automatic, being able to simply scan for and replace the library you know is bad is a heck of a lot easier then recompiling a statically linked exe.
People are okay with really specific dependencies when it's part of the business critical application they're supporting - i.e. the nodejs or python app which runs the business, that can do anything it wants we'll keep it running no matter what. Having this happen to the underlying distributions though?
(of note: I've run into this issue with Go - love the static deploys, but if someone finds a vulnerability in the TLS stack of Go suddenly we're rushing out rebuilds).
Is that ultimately the responsibility of the application developer or the OS developer?
The issue is one way or another it needs to happen ASAP: so either the distro is haranguing upstream to "approve" a change, or they're going to be going in and carrying patches to Cargo.toml anyway - so this idea of "don't you dare touch my dependencies" lasts exactly as long as until you need a fix in ASAP.
And even if they are, for rust software that doesn't come from debian, there is no way to ensure it all gets rebuilt and updated with the fix.
Also, projects are generally slow (taking several months) to accept patches. When a distribution has fixed something and the users notice no issue, the upstream project if downloaded and compiled would be a different matter entirely.
In C, it is generally assumed that libraries maintain compatibility within one major version, so programs rarely have tight version intervals and maintainers could just use the newest available version (in each major series) for all packages depending on it.
If the build system (for Rust and some other languages) makes it easy to depend on specific minor/patch versions (or upper-bound intervals of versions), it encourages developers to do so instead of working on fixing the mess in the ecosystem.
openssl has made several rounds of incompatible changes. ffmpeg and related av* libs make major releases frequently. libx264 is famous for having "just pin a commit" approach to releases.
It's common for distros to carry multiple versions of major libraries. Sometimes it's a frequent churn (like llvm), sometimes it's a multi-year migration (gtk, ncurses, python, libpng).
C libraries aren't magically virtuous in their versioning. The language doesn't even help with stability, it's all manual painstaking work. You often don't see these pains, because distro the maintainers do heroic work of testing the upgrades, reporting issues upstream, holding back breaking changes, and patching everything to work together.
----
Cargo packages almost never pin specific dependency versions (i.e. they don't depend on exact minor/patch version). Pinning is discouraged, because it works very poorly in Cargo, and causes hard dependency resolution conflicts. The only place where pinning is used regularly is pairs of packages from the same project that are expected to be used together (when it's essentially one version of one package, but had to be split into two files for technical reasons, like derive macros and their helper functions).
By default, and this is universally used default, Cargo allows semver-major-compatible dependency upgrades (which is comparable to sover). `dep = "1.2.3"` is not exact, but means >=1.2.3 && <2.0.0. The ecosystem is quite serious about semver compatibility, and there is a tooling to test and enforce it. Note that in Cargo the first non-zero number in the version is the semver-major.
I release foo 1.2.3 as a package, when doing it depending on bar 4.5.6. Perhaps foo still compiles with bar 4.5.2 or 4.6.8... but that's _not_ foo 1.2.3 right? And if there are bug fixes in some of those but not others, if I had my deps set up to just package with "bar 4.x" , suddenly I'm getting bug reports for "1.2.3" without a real association to "the" 1.2.3.
I'm saying this, perhaps this is as easy as having two sets of deps (like deps for repackagers, deps for people wanting to reproduce what I built)... but if a binary is being shipped _not_ being explicit about the exact dependencies used is upstream of a whole lot of problems! Is there a great answer from OS distro package managers for this?
Is the thing that should happen here be that there's a configure step in rust builds that would mess around with deps based on what you have?
Do you rewrite every API call in your software every time you bump dependencies?
Do you think that a library that changes API very often and thus remains insecure because bumping it breaks the program should be running on people's computers?
If it has a 0.x.y version number? Yes, that's what 0.x means to begin with. In fact Rust/cargo provides a stricter interpretation of semver than the original, which allows you to express guarantees of API stability (by keeping the minor version number fixed) even in 0.x-version projects.
It was a mess, and linux package managers were first to actually solve the problem. I fully believe that the explosive growth of linux in the early years had little to do with the kernel, and was much more about the proper package managers introduced by Debian, Red Hat, and friends.
But just because package managers were massively better than nothing, and nothing (universally) better has come along since, doesn't mean they are actually good. The user experience is pretty good. But for developers, the drive to dynamically link every single dependency, and only have one (maybe two) versions of each dependency, and to randomly update dependencies causes massive problems for developers. It only works as well as it does for C/C++ applications because they have gotten used to working within the restrictive framework. And maybe a bit of Stockholm syndrome.
-----------------
Let me tell you a story:
Back in 2014, I was working on Dolphin Emulator (which is entirely c++), trying to fix regressions for a Dolphin 5.0 release. One of these regressions was that on linux, launching a game a second time (or opening the controller config window twice) would cause a crash. I quickly tracked the bug down to a bug introduced in SDL 2.0.3. Most consumers of SDL weren't impacted by this bug, because they only ever initialised SDL once.
The fix was simple enough, it had already been fixed in the latest reversion and so would be fixed whenever 2.0.4 was released. But we wanted to release before that, so how to fix the bug? If this was cargo, I would just point it at a git revision (possibly a custom branch, with just the fix back ported). But that wasn't an option.
I spent ages trying to work out a workaround that avoided the bug, but couldn't find one. I considered including a patched version of SDL with dolphin; Dolphin's source tree already contained a bunch of libraries which we statically linked in. Some patched, others were just there for easier builds on Windows/Mac). But I knew from experience that any distro would simply patch our build system to use the distro version. Dolphin's build system already had an extensive set of manual checks to use the distro version of a library if it existed. Many of these of these checks were contributed upstreams by distro package maintainers.
I considered throwing an error message (twice, both at build and runtime) if dolphin was linked with a bugged version of SDL 2... But I figured the distro maintainers would probably patch out both error message too. Remember, we are talking about a bug that wouldn't even be fixed to the next release of SDL, with an uncertain release date. And there would probably be distros which stuck with the buggy version for way too long (Debian actually avoided the bug, because it was still shipping SDL 2.0.1, before the bug was introduced)
Eventually I got frustrated. And I realised that SDL was only used for controller input, and only on linux/bsd. SDL already wasn't meeting our needs, we had a dedicated core input backend for MacOS and both DInput and XInput backends for Windows because they worked better. I had also just gone though the entire SDL input code, back when I was trying to find a workaround, and had learned how it work.
So... to workaround the limitations of Linux Disto packaging (and a stupid, easy to fix SDL bug), I simply deleted SDL out of the Dolphin codebase. Only took me three days to completely reimplement all the controller input functionality, directly accessing the lower-level linux libraries: libudev for controller detection/hotpugging, open the correct `/dev/input/eventX` file, libevdev for parsing the raw input events.
Three days to implement it. A few weeks of testing to find bugs... Not only did this workaround the bug. But it allowed Dolphin to access controller functionality that SDL wasn't exposing at the time (suddenly, the Dual Shock 4's touchpad would work, and the pressure sensitive buttons on Dual Sock 3). I don't think SDL was even exposing the gyroscope/accelerometers of those controllers at the time.
---------------------
I was very happy with the result, but it was crazy that I was forced to go to such extremes, just because of the linux package manager insistence on enforcing one shared version of a dependency. I'm not sure how common it is for developers to rip-out and replace an entire dependency because of distro packaging, but workarounds for buggy library versions are common.
These days, I'm really enjoying programming with rust; The fact that cargo always gives me the exact version of the package I asked for is a massive win. And why should we give it up, just because distros want to do something worse. And it would be so hard to make the transition, because a lot of rust code has been written under the assumption that cargo will give you the version (or version range) than you ask for.
If you think about it, it's kind of insane that packages will be build and tested against one version of a library, just to have it randomly replaced with a later version, potentially introducing bugs. In some edge cases, packages even get downgraded to an older version between build/test and being installed on a user's system.
---------------------
What's the solution for rust? I have no idea. Distros do have a point that statically linking everything isn't the best idea. Maybe we can reach some kind of middle ground where some dependencies are statically linked and others are dynamically linked, but with explicit support for multiple versions of each dependency co-existing on the system.
Maybe we also add explicit support to package managers to track which dependencies have been statically linked into them, so we can force a rebuild of them all when there is a security issue.
Distro maintainers will not patch in a bug into a program, they will patch out bug from the SDL. :-/
But I still would have required an interim solution for everyone building from source until these patched/fixed versions of SDL started shipping in distros. And then how would the warning message tell the difference between the original SDL 2.0.3 and a patched version of SDL 2.0.3?
And we are talking about dozens of distros, multiple chances for something to go wrong.
My point is that I quickly came to the conclusion that ripping out SDL and reimplementing the functionality was the simplest, most conclusive solution.
$ rpm -q SDL2
SDL2-2.30.9-1.fc40.x86_64
SDL2-2.30.9-1.fc40.i686
This to me sounds like the distro packaging model isn’t a good one.
I wish people would interrogate this more deeply: why do we see so many Dockerfiles with random binary artifacts? Why do people download mystery meat binaries from GitHub releases? Why is curl-pipe-to-bash such a popular pattern?
The answer to the questions is that, with few exceptions, distributions are harder to package for, both in first- and third-party contexts. They offer fewer resources for learning how to package, and those resources that do exist are largely geared towards slow and stable foundational packaging (read: C and C++ libraries, stable desktop applications) and not the world of random (but popular) tools on GitHub. Distribution packaging processes (including human processes) similarly reflect this reality.
On one level, this is great: like you, I strongly prefer a distro package when it's available, because I know what I'm going to get. But on another level, it's manifestly not satisfying user demand, and users are instead doing whatever it takes to accomplish the task at hand. That seems unlikely to change for the better anytime soon.
Distributions using a "model of global shared libraries".
If you have to run 5 different docker images each with their own “global shared library” set you clearly no longer have system wide globals. You have an island of deps potentially per program or possible a few islands for a few sets of programs.
Which, once again, completely defeats the entire purpose of the Linux global shared library model. It would have been much much much simpler for each program to have linked statically or to expect programs to include their dependencies (like Windows).
Containers should not exist. The fact that they exist is a design failure.
If you’re going to say something so spicy, at least back it up with some reasons, otherwise you could have just said it out loud and saved everyone from reading it (blah blah HN guidelines blah blah).
Naw. Lack of reliability in running programs is a distinctly Linux phenomenon. Windows culture is much much much better.
I know I’m dealing with programmers but the level of precision required on internet comments is absurd.
I work on Windows and don’t use Docker so I too known it’s not Everyone. But when I said everyone I would like to think not Everyone assumed I meant literally Everyone. Capitalization intended.
If you compile my Rust library against dependencies that are not compatible as declared in Cargo.toml, the result is on you. If you want to regenerate a compatible Cargo.lock, that would be more understandable, and I don't go out of my way to specify "=x.y.z" dependencies in my Cargo.toml, so I have effectively given permission for that anyway.
They give the example of Debian "bumping" dependencies, but given the relative "freshness" of your typical Rust package vs say... Debian stable, I imagine the more likely outcome would be downgrading dependencies.
This reminds me of the time Debian developers "knew better" with the openssh key handling...
> This reminds me of the time Debian developers "knew better" with the openssh key handling...
How much % of the CVEs that exists are from debian patches and how much is upstream developers? Can you confidently claim that upstream developers are better at security than debian developers? Or are you just picking a single example and ignoring all the rest of the data that you don't like?
I have 3794 packages currently installed on my machines. If all of them required special handling like your snowflake library I'd just not be using a computer at all, because I certainly don't have the capacity to deal with 3794 special snowflakes.
It's hardly unique. Every major language other than C/C++ (and Python, but Python is just a mess in general) works like Rust here. If anything I'd say it's the C people who are the snowflakes.
> Can you confidently claim that upstream developers are better at security than debian developers?
I can confidently claim that upstream has better policies. Debian routinely patches security-critical software without dedicated security review; this predictably lead to disaster in the past and will lead to disaster again.
Also the Debian SSL key bug was not just one more CVE on a par with any other. It was probably the worst security bug ever seen in general-purpose software.
This is untrue. Provide some evidence or withdraw your false claim. Thanks.
Debian has thousands of different upstream, you don't even know who they are, how can you claim to know their policies and to have evaluated all of them?
Also, all the 1 person projects do not get fixes in a timely manner. As you would understand if there was any amount of reasoning behind your arguments.
Are you done making stuff up or must you continue?
> Also the Debian SSL key bug
1. Shit happens
2. If you want random data you read from /dev/random, an uninitialised buffer is NOT random.
3. If your example dates to over a decade ago, I'd say the track record is excellent.
It's a good reason for others to avoid your libraries, though, especially if they do intend to be compatible with common distros.
The openssl issue was just the result of openssl being unnecessarily complex and difficult to work with, though.
This is a totally unfair comparison. The upstream OpenSSL code was broken: it was invoking undefined behaviour. Moreover, the Debian maintainer had asked upstream whether it was OK to remove the code, because it was causing Valgrind memory warnings.
No it wasn't. It was reading indeterminate character values, which has defined behaviour (char cannot have a trap representation).
> the Debian maintainer had asked upstream whether it was OK to remove the code
They hadn't asked the correct mailing list, and they only asked at all for one of the two cases where they removed similar code, which wasn't the one taht caused the problem.
That's not how Debian development is done, fresh software is uploaded to unstable only and then eventually ends up in a stable release. As a maintainer, Debian stable is something you support, not something you develop against.
The annoying thing is that that was always supposed to be true. Users of distro packages are already supposed to report all bugs to the package maintainers, and then they deal with it. Sometimes part of dealing with it is finding that a bug did in fact originate upstream and reporting it, but that was never supposed to be step 1.
So "user reporting a bug" is already an unusually highly motivated individual compared to the alternatives. Which leads to (3) they were trying to use the software right then, and a response from package maintainers of "this is an upstream bug" doesn't solve the problem, and in fact barely implies that a solution might be being requested from upstream.
Which also leads to (4): the most common immediate resolution anyone is going to present that user is "use the latest version / master / whatever". Which means the user is already going to make the transition to "now I'm a direct upstream user" (and equally the high probability the bug might still exist in upstream anyway).
I only read the comments here so far and not the article but it sounds to me that people are up-in-arms about the "how do we handles" related to:
* primarily C/GNU makefile ecosystem
then
you now want to add Rust to it
and
Cargo.lock on cares about Cargo.toml versions, not what other crates `build.rs` are linking against provided by the system external to Rust?
> I am proposing that Debian should routinely compile Rust packages against dependencies in violation of the declared semver, and ship the results to Debian’s millions of users.
Followed by a rationale. However, the rationale rings hollow to me:
* Compiling a Rust package outside of its declared semver might not cause security problems in the form of exploitable memory corruption, but it's almost certainly going to cause stability problems with extremely undesirable characteristics (impossible for the upstream to triage, and definitionally unsupported).
* The assumption that semantic changes to APIs are encoded within types is incorrect in the general case: Rust has its fair share of "take a string, return a string" APIs where critical semantic changes can occur without any changes to the public interface. These again are unlikely to cause memory safety issues but they can result in logical security issues, especially if the project's actual version constraints are intended to close off versions that do work but perform their operation in an undesirable way.
As a contrived example of the above: `frobulator v1.0.0` might have `fn frob(&str) -> &str` which shells out to `bash` for no good reason. `frobulator v1.0.1` removes the subshell but doesn't change the API signature or public behavior; Debian ends up using v1.0.0 as the build dependency despite the upstream maintainer explicitly requesting v1.0.1 or later. This would presumably go unnoticed until someone files a RUSTSEC advisory on v1.0.0 or similar, which I think is a risky assumption given the size (and growth) of the Rust ecosystem and its tendency for large dep trees.
The author is right that, in practice, this will work 99% of the time. But I think the 1% will cause a lot of unnecessary downstream heartburn, all because of a path dependency (assumptions around deps and dynamic linkage) that isn't categorically relevant to the Rust ecosystem.
(Problems along this line already are why the bcachefs dev told users to avoid debian and the packager eventually dropped the package: https://www.reddit.com/r/bcachefs/comments/1em2vzf/psa_avoid...)
At the same time, I suspect this kind of policy will make my life harder than necessary: users who hit bugs due to distribution packaging quirks will be redirected to me for triage, and I'll have a very hard time understanding (much less fixing) quirks that are tied to transitive subdependencies that my own builds intentionally exclude.
This was my thought. Xubuntu decided to ship unstable development pre-releases of the core Xfce components in their last stable OS release, and I got really annoyed getting bug reports from users who were running a dev release that was 4 releases behind current, where things were fixed.
This was completely unnecessary and wasted a bunch of my time.
The problem is users have gotten savvy about talking to upstream directly. It helps that upstream is usually on easy-to-use and search-engine-indexed platforms like GitHub these days vs mailing lists. It would still be fine if the user did the diligence of checking whether the problem exists in upstream or just the distro package, but they often don't do that either.
I like Debian's stability and don't really need my filesystem to move fast and break things. If this is the direction bcachefs is going, I don't think it's ever going to be a good platform to run an OS on top of.
Related HN discussion from August/September: https://news.ycombinator.com/item?id=41407768
This is a completely different situation from C/C++ and from building a distro out of C/C++ source. C does not have a useful package manager, and it’s handy if my distro has C libraries packaged. My distro might use them!
(I might also use them for my own unpackaged work! But then I might get grumpy that the result of building my own code isn’t actually useful outside that version of that distro, and then I risk having to use containers (sigh), when what I actually wanted was just a build artifact that could run on other distros.)
If the above is right this contortion is similar to what's happened with Python packaging in Debian and similar, where distributions tried hard to maintain compatibility inside of a single global environment instead of allowing distinct incompatible resolutions within independent environments (which is what Python encourages).
I think the "problem" with cargo-deb is that it bundles all-in-one, i.e. doesn't devolve the dependencies back to the distribution. In other words, it's technically sound but not philosophically compatible with what Debian wants to do.
If I was paid to do this kind of useless work for a living I’d probably be well on the way to off myself.
In that context, creating an ever increasing menagerie of dependencies of subtly different versions of the same thing is the pointless busy work because for the tiny bit of developer convenience, you're opening an exponentially growing amount of audit work to prove the dependencies are safe.
But I'd note that a hypothetical "verified NPM" would also result in the same thing: Microsoft does not have infinite resources for such a thing, so you'd just have a limited set of approved deps yet again (which would in fact make piggy backing them relatively easy for distros).
I can't see a way to slice it where it's reasonable to expect such a world to just support enormous nests of dependency versions.
I'd expect it to be much less effort for Debian to use the dependencies the application says it's compatible with, and build tooling allowing them to track/audit/vendor/upgrade Cargo dependencies as needed; rather than trying to shove the Cargo ecosystem into their existing tooling that is fundamentally different (and thus incompatible).
But, if you like it, find like-minded people here on HN and create a successful distro with static linking only. Maybe you will success where others failed. Thank you in advance.
Edit: I explained lower down but I also want to mention here, static linkage of binaries is a huge burden and waste of resources for a Linux distro. That's why they all tend to lean heavily on shared libraries unless it is too difficult to do so.
I don't think any of this precludes immutability: my understanding is Debian could package every version variant (or find common variants without violating semver) and maintain both immutability and their global view. Or, they could maintain immutability but sacrifice their package-level global view (but not metadata-level view) by having Debian Rust source packages contain their fully vendored dependency set.
The former would be a lot of work, especially given how manual the distribution packaging process is today. The latter seems more tractable, but requires distributions to readjust their approach to dependency tracking in ecosystems that fundamentally don't behave like C or C++ (Rust, Go, Python, etc.).
> How bad would it suck if every Python app you installed needed to have its own venv for example?
Empirically, not that badly. It's what tools like `uv` and `pipx` do by default, and it results in a markedly better net user experience (since Python tools actually behave like hermetic tools, and not implicit modifiers of global resolution state). It's also what Homebrew does -- every packaged Python formula in Homebrew gets shipped in its own virtual environment.
> A properly maintained global environment can do that.
Agreed. The problem is the "properly maintained" part; I would argue that ignoring upstream semver constraints challenges the overall project :-)
That would work for distributions that provide just distributions / builds. But one major advantage of Debian is that it is committed to provide security fixes regardless of upstream availability. So they essentially stand in for maintainers. And to maintain many different versions instead of just latest one is plenty of redundant work that nobody would want to do.
That's not reasonable for library packages, because they may have to interact with each other. You're also proposing a scheme that would cause an explosion in resource usage when it comes to compilation and distribution of packages. Packages should be granular unless you are packaging something that is just too difficult to handle at a granular level, and you just want to get it over with. I don't even know if Debian accepts monolithic packages cobbled together that way. I suspect they do, but it certainly isn't ideal.
>And to maintain many different versions instead of just latest one is plenty of redundant work that nobody would want to do.
When this is done, it is likely because updating is riskier and more work than maintaining a few old versions. Library authors that constantly break stuff for no good reason make this work much harder. Some of them only want to use bleeding edge features and have zero interest in supporting any stable version of anything. Package systems that let anyone publish easily lead to a proliferation of unstable dependencies like that. App authors don't necessarily know what trouble they're buying into with any given dependency choice.
Debian is a binary-first distro so this would obligate them to produce probably 5x the binary packages for the same thing. Then you have higher chances of conflicts, unless I'm missing something. C and C++ shared libraries support coexistence of multiple versions via semver-based name schemes. I don't know if Rust packages are structured that well.
>Empirically, not that badly. It's what tools like `uv` and `pipx` do by default, and it results in a markedly better net user experience (since Python tools actually behave like hermetic tools, and not implicit modifiers of global resolution state). It's also what Homebrew does -- every packaged Python formula in Homebrew gets shipped in its own virtual environment.
These are typically not used to install everything that goes into a whole desktop or server operating system. They're used to install a handful of applications that the user wants. If you want to support as many systems as possible, you need to be mindful of resource usage.
>I would argue that ignoring upstream semver constraints challenges the overall project :-)
Yes it's a horrible idea. "Let's programmatically add a ton of bugs and wait for victims to report the bugs back to us in the future" is what I'm reading. A policy like that can be exploited by malicious actors. At minimum they need to ship the correct required versions of everything, if they ship anything.
Ah yeah, this wouldn't work -- instead, Debian would need to bite the bullet on Rust preferring static linkage and accept that each package might have different interior dependencies (still static and known, just not globally consistent). This doesn't represent a conflict risk because of the static linkage, but it's very much against Debian's philosophy (as I understand it).
> I don't know if Rust packages are structured that well.
Rust packages of different versions can gracefully coexist (they do already at the crate resolution level), but static linkage is the norm.
> These are typically not used to install everything that goes into a whole desktop or server operating system. They're used to install a handful of applications that the user wants.
I might not be understanding what you mean, but I don't think the user/machine distinction is super relevant in most deployments: in practice the server's software shouldn't be running as root anyways, so it doesn't matter much that it's installed in a user-held virtual environment.
And with respect to resource consumption: unless I'm missing something, I think the resource difference between installing a stack with `pip` and installing that same stack with `apt` should be pretty marginal -- installers will pay a linear cost for each new virtual environment, but I can't imagine that being a dealbreaker in most setups (already multiple venvs are atypical, and you'd have to be pretty constrained in terms of storage space to have issues with a few duplicate installs of `requests` or similar).
Many software packages need root access but that is not what I was talking about. Distro users just want working software with minimal resource usage and incompatibilities.
>Rust packages of different versions can gracefully coexist (they do already at the crate resolution level), but static linkage is the norm.
Static linkage is deliberately avoided as much as possible by distros like Debian due to the additional overhead. It's overhead on the installation side and mega overhead on the server that has to host a download of essentially the same dependency many times for each installation when it could have instead been downloaded once.
>And with respect to resource consumption: unless I'm missing something, I think the resource difference between installing a stack with `pip` and installing that same stack with `apt` should be pretty marginal -- installers will pay a linear cost for each new virtual environment, but I can't imagine that being a dealbreaker in most setups (already multiple venvs are atypical, and you'd have to be pretty constrained in terms of storage space to have issues with a few duplicate installs of `requests` or similar).
If the binary package is a thin wrapper around venv, then you're right. But these packages are usually designed to share dependencies with other packages where possible. So for example, if you had two packages installed using some huge library for example, they only need one copy of that library between them. Updating the library only requires downloading a new version of the library. Updating the library if it is statically linked requires downloading it twice along with the other code it's linked with, potentially using many times the amount of resources on network and disk. Static linking is convenient sometimes but it isn't free.
This hasn't been true for decades now.
Plugins and OS extensions were also a reason why they came to be.
Static linking works fine because 99% of what you run is dynamically linked.
Try to statically link your distribution entirely and see how the ram usage and speed will degrade :)
For python, you could install libraries into a versioned dir, and then create a venv for each program, and then in each venv/lib/pythonX/site-packages/libraryY dir just symlinks to the appropriate versioned global copy.
unlike pypi, debian patches CVEs, so having 3000 copies of the same vulnerability gets a bit complicated to manage.
Of course if you adopt the pypi/venv scheme where you just ignore them, it's all much simpler :)
* Comparing the two in this regard is a category error: Debian offers a curated index, and PyPI doesn't. Debian has a trusted set of packagers and package reviewers; PyPI is open to the public. They're fundamentally different models with different goals.
* PyPI does offer a security feed for packages[1], and there's an official tool[2] that will tell you when an installed version of a package is known to be vulnerable. But this doesn't give PyPI the ability to patch things for you; per above, that's something it fundamentally isn't meant to do.
One project patches/updates vulnerable software and makes sure everything else works, while the other puts all the effort on the user.
You mean…the way many modern python apps install themselves? By setting up their own venv? Which is a sane and sensible thing to do, given pythons sub-par packaging experience.
The problem extends way beyond Python. This is why we have Docker, Snap, Flatpack, etc.: to work around inadequate maintenance and package conflicts without touching any code. These tools make it even easier for package authors to overlook bad habits. "Just use a venv bro" or "Just run it in Docker" is a common response to complaints.
I want to challenge it, because I’m beginning to be of the opinion that the “distro way” isn’t actually entirely suitable for a lot of software _anymore_.
The fact that running something via docker is easier for people than the distro way indicates that there are significant UX/usability issues that aren’t being addressed. Docker as a mechanism to ship stuff for users does suck, and it’s a workaround for sure, but I’m not convinced that all those developers are “doing things wrong”.
It's a lack of maintenance that isn't getting addressed. Either some set of dependencies isn't being updated, or the consumers of those dependencies aren't being updated to match changes in those dependencies. In some cases there is no actual work to do, and the container user is just too lazy to test anything newer. Why try to update when you can run the old stuff forever, ignoring even the slow upgrades that come with your Linux distro every few years?
Some people would insist that running old stuff forever in a container is not "doing things wrong" because it works. But most people need to update for security reasons, and to get new features. It would be better to update gradually, and I think containers discourage gradual improvements as they are used in most circumstances. Of course you can use the technology to speed up the work of testing many different configurations as well, so it's not all bad. I fear there are far more lazy developers out there than industrious ones however.
I would love to have that. Actually that's what I do: I avoid distribution software as much as possible and install it in venvs and similar ways.
You just described every python3 project in 2024. Pretty much none will be expected to work with system python. But your point still stands, that's not a good thing that there is no python and only pythons. And it's not a good thing that there is no rustc only rustcs, etc, let alone trying to deal with cargo.
(This is distinct from Rust, where there’s no global package namespace at all.)
Yeah I hacked together a shim that searches the python program's path for a directory called venv and shoves that into sys.path. Haven't hacked together reusing venv subdirs like pnpm does for JavaScript, but that's on my list.
Does it? Even if 2 of those chromiums and all of the QTs have actively exploited vulnerabilities and it's anyone's guess if/when the application authors might bother updating?
This is clearly a tradeoff with no easy win for either side.
Could you remind those of us who don't remember that? The big one I know about is CVE-2008-0166, but that was 1. not a backport, and 2. was run by upstream before they shipped it.
But yes, agreed that it's painful either way; I think it comes down to that someone has to do the legwork, and who exactly does it has trade-offs either way.
Regardless, the original point was:
>>That version may well be out of support and not receive fixes at all any more, which leaves the burden of maintenance on the distribution.
... and my response that this is only a problem for LTS distros stands. A rolling release distro will not get in the business of maintaining packages after upstream dropped support. It is only done by LTS distros, because the whole point of LTS distros is that their packages must be kept maintained and usable for N years even as upstreams lose interest and the world perishes in nuclear fire and zombies walk the Earth feasting on the brains of anyone still using outdated software.
---
Now, to play devil's advocate, here's an OpenSUSE TW (rolling distro) bug that I helped investigate: https://bugzilla.opensuse.org/show_bug.cgi?id=1214003 . The tl;dr is that:
- Chromium upstream vendors all its dependencies, but OpenSUSE forces it to compile against distribution packages.
- Chromium version X compiles against vendored library version A. OpenSUSE updates its Chromium package to version X and also has library version A in its repos, so the Chromium compiled against distro library works fine.
- Chromium upstream updates the vendored library to version B, and makes that change as part of Chromium version Y. OpenSUSE hasn't updated yet.
- OpenSUSE updates the library package to version B. Chromium version X's package is automatically rebuilt against the new library, and it compiles fine because the API is the same.
- Disaster! The semantics of the library did change between versions A and B even though the API didn't, so OpenSUSE's Chromium now segfaults due to nullptr deref. Chromium version Y contains Chromium changes to account for this difference, but the OpenSUSE build of Chromium X of course doesn't have them.
You will note that this is caused by a distro compiling a package against a newer version of a dependency than upstream tested it against, not an older one, but you are otherwise welcome to draw parallels from it.
In this case it was fixed by backporting the Chromium version Y change to work with library version B, and eventually the Chromium package was updated to version Y and the patch was dropped. In a hypothetical scenario where Chromium could not be updated nor patched (say the patch was too risky), it could have worked for the distro to make the library coinstallable and then have Chromium use library version A while everything else uses version B.
Or, as another framing: everything about user-side packaging cuts both ways: it's both a source of new dependency and vulnerability tracking woes, and it's a significant accelerant to the process of getting patched versions into place. Good and bad.
s/dependencies/security holes/g
Because that works so well on windows right?
Rust programs have fewer CVEs for two reasons: its safe design, and its experienced user base. As it grows more widespread, more thoughtless programmers will create insecure programs in Rust. They just won’t often be caused by memory bugs.
Thankfully, we have better data sources. Chromium estimates that 70% of serious security bugs in their codebases stem from memory unsafety[1], and MSRC estimates a similar number for Microsoft's codebases[2].
(General purpose programming languages can't prevent logic bugs. However, I would separately argue that idiomatic Rust programs are less likely to experience classes of logic bugs that are common in C and C++ programs, in part because strong type systems can make invalid states unrepresentable.)
[1]: https://www.chromium.org/Home/chromium-security/memory-safet...
[2]: https://msrc.microsoft.com/blog/2019/07/we-need-a-safer-syst...
Some users commenting here are employed by python, which also has a policy of breaking compatibility all the time.
It's very fun to develop in this way, but of course completely insecure.
These developers might not care because they aren't in charge of security, while distributions care about security and distro maintainers understand that shipping thousands of copies of the same code means you can't fix vulnerabilities (and it's also terribly inefficient).
My suspicion is that most developers think their own software is a special snowflake, while everyone else's software is still expected to behave normally because who's got the time to deal with that anyway?
The distinction between consultant and employee is pretty moot if you're not the tax office.
It's a bold move, Cotton.
Wiping `Cargo.lock` and updating within semver seems sort-of reasonable if you're talking about security updates like for a TLS lib.
"Massaging" `Cargo.toml` and hoping that automated tests catch breakage seems doomed to fail.
Of course this was 3 years ago and this person is no fool. I wonder what I'm missing.
Did this ever go into production?
For unclear reasons it seems Debian only packages the sources and not the binaries and thus doesn't need the Rust version, and also apparently only packages the latest semver version (which might make it problematic to compile old versions but is not an issue at runtime).
Aren’t emerge and other systems in source based distributions like gentoo doing this for 20 odd years now ? https://devmanual.gentoo.org/general-concepts/dependencies/
I don’t understand how the RDEPEND , BDEPEND setup doesn’t handle this, and the problem should be hardly unique for just rust ?
Perhaps handling this in the package layer in a distribution not designed to handle build dependencies properly is the problem ?
Isn’t this what Go’s MVS does? Or am I misunderstanding?
>= doesn't really mean anything below it will break
However, I think programming and the ecosystem is different these days and I am beginning to think the “single globally shared dylibs” model isn’t a good fit for many of the more recent programming languages and ecosystems.
We’ve seen it with the bcacheFS stuff and I think we’ll see it with more stuff in the future. I suspect there’s fundamental culture differences.
As mentioned elsewhere in the thread, I’d be interested in a distro model that split out “OS” and “app” level management and does not concern itself with “user”/“app” level dependencies, beyond maybe tracking the installed versions of apps, and little else.
That sounds like a recipe for a lot more security vulnerabilities like https://news.ycombinator.com/item?id=30614766
This means if you change my chosen dependencies you are responsible to ensure testing exists.
Unless debian packagers provide the tests to avoid logic errors