The successful languages without efficient dependency management are painful to manage dependencies in, though. I think Python should be shooting for a better package management user experience than C++.

If Python's dependency management is better than anything, it's better than C++'s. Python has pip and venv. C++ has nothing (you could say less than nothing since you also have ample opportunity for inconsistent build due to mismatching #defines as well as using the wrong binaries for your .h files and nothing remotely like type-safe linkage to mitigate human error. It also has an infinite number of build systems where each system of makefiles or cmakefiles is its own build system with its own conventions and features). In fact python is the best dependency management system for C++ code when you can get binaries build from C++ via pip install...

> If Python's dependency management is better than anything, it's better than C++'s.

That’s like the lowest possible bar to clear.

Agreed, but that was the bar set by the comment I was replying to, which claimed Python doesn't clear it.

On bro

C++ has apt-get etc. because the libraries do not change all the time. Also, of course there are vcpkg and conan.

Whenever you try to build something via pip, the build will invariably fail. The times that NumPy built from source from PyPI are long over. In fact, at least 50% of attempted package builds fail.

The alternative of binary wheels is flaky.

> C++ has apt-get

That's not a development dependency manager. System package management is a different kind of issue, even if there's a bit of overlap.

> because the libraries do not change all the time

That's not true in practice. Spend enough time with larger projects or do some software packaging and you'll learn that the pain is everywhere.

That was the entire point, that C++ is the absolute worst.

I pip3 installed something today. It didn’t work, at all.

I then yum installed a lib and headers, it worked well.

C++ on an msft platform is the worst. I can’t speak for Mac. C++ on a linux is quite pleasant. Feels like most of the comments like yours are biased for un-stated reasons.

This has nothing to do with languages. You can yum install python packages and expect them to work fine. You can install C++ files using an actual dependency manager like vcpkg or conan and have issues.

You're pointing out differences between software package management styles, not languages.

C++ on linux is indeed pleasant if you use only distro-provided library versions. Some specialized library with specific version, also no big deal. Need some upgraded version of a widely-used library -- get containerized or prepare for real pain.

If I had a penny for every time I gave up on compiling C++ software because there's no way to know what dependencies it needs, I'd be a millionaire. Python at least lists them.

Is that because the compiler failed with "foo.h not found" or the build system said "libfoo not found"? CMake is most common and it will tell you. Worst case it's difficult to derive the package name from the name in the diagnostic.

It's not great, but usually not a big deal neither IME. Typically a couple of minutes to e.g. find that required libSDL2 addon module or whatever, if there is that kind of problem at all.

Yes it is, and it's usually such a big deal for me that I just don't use that software. I don't have time to go through a loop of "what's the file name? What package is it in? Install, repeat". This is by far the worst experience I've had with any language. Python has been a breeze in comparison.

I’m not going to refute your points. If you’re going to wear rose-tinted glasses about all of the bad parts about python, that’s fine, I also like python.

What's rose-tinted about "one of them downloads dependencies automatically, the other one doesn't"?

I mean this is a documentation problem. It's pretty common for python to import something it doesn't say it depends on too, btw...

If I had a penny every time I heard something like that on sites like this, I’d be a billionaire :)

Have you tried vcpkg on msft (work on linux and mac too btw)? I found it to be much better than pip3 and venv nonsense.

Mac has the Brew project, which is sort of like apt-get or yum.

Python's dependency management sucks because they're audacious enough to attempt packaging non-python dependencies. People always bring Maven up as a system that got it right, but Maven only does JVM things.

I think the real solution here is to just only use python dependency management for python things and to use something like nix for everything else.

Julia's package manager (for one) works great and can manage non Julia packages. the problem with python's system is that rejecting semver makes writing a package manager basically impossible since there is no way to automatically resolve packages.

Could you clarify what you mean? pip and every other Python package installer is absolutely doing automatic package resolution, and the standard (PEP 440) dependency operators include a compatible version operator (~=) that's predicated on SemVer-style version behavior.

There is no way to manage non-hosted dependencies, though, in a cross-platform way. Something attempting it is often worse than nothing, on a distro that has different assumptions — e.g. every package manager that downloads a dynamic executable will fail on NixOS, and gives no easy way to hook into how those get executed.

I agree that attempting it is worse than nothing, because you now have expectations that may fail at awkward times. But they've gone an done it so here we are.

NixOS is a stark contrast to Python here. It makes things that can't be done deterministically difficult to do at all. Maybe this sounds extreme from the outside, but I'd rather be warned off from that dependency as soon as I attempt to use it, rather than years later when I get a user or contributor than can't make it work for some environmental reason I didn't forsee and now everything rests on finding some hacky way to make it work.

If Nix can be used to solve Python's packaging problems, participating packages will have to practice the same kind of avoidance (or put in the work to fix such hazards up front). I'm not sure if the wider python community is willing to do that, but as someone who writes a lot of python myself and wants it to not be painful down the line, I am.

This is what we used to have and it was much worse. Source: lived that life 10-15 y ago.

15y ago I was using apt-get to manage my c++ dependencies with no way of keeping track of which dependency went with which project. It was indeed pretty awful.

Now when I cd into a project, direnv + nix notices the dependencies that that project needs and makes them available, whatever their language. When I cd into a different project, I get an entirely different set of dependencies. There's pretty much nothing installed with system scope. Just a shell and an editor.

Both of these are language agnostic, but the level of encapsulation is quite different and one is much better than that other. (There are still plenty of problems, but they can be fixed with a commit instead of a change of habit.)

The idea that every language needs a different package manager and that each of those needs to package everything that might my useful when called from that language whether or not it is written in that language... It just doesn't scale.

Valid points. I was also thinking about apt install-ing some C library, try to pip install a package, have it fail to build, look for headers/debug packages in apt, set LD_LIBRARY… you know. It hurts, a lot.

And yet, python is a fantastic language because it’s the remote control to do the heavy, complex, low-level, high-performance stuff with relative ease.

I agree. I want to help save python from its packaging problems because it has so much traction out there (for good reason). People in this forum might be ok jumping ship to rust or whatever but most python users are students, scientists, business types... They're not going to learn rust and I don't think we benefit from leaving them behind.

I wish there was a standard interface that tools like pip could use to express their non-python needs such that some other tool can meet those needs and then hand the baton back to pip once they are met. Poetry2nix is an example of such a collaboration. (I'm not trying to be a nix maximalist here, it's just that it's familiar).

The python community is large enough to attempt to go it alone, but many other language communities are not. I think we'd see faster language evolution if we asked less of them from a packaging perspective:

> focus on making the language great. Provide a way to package deps in that language, and use _____ to ask for everything else.

Apt and brew and snap and whatever else (docker? K8s?) could be taught to handle such requests in a non alter-your-whole-system kind of way.

Not sure this is still a valid critic of Python in 2024.

Between pip, poetry and pyproject.toml, things are now quite good IMHO.

I guess that depends from your perspective. I'm not a Python developer, but like many people I do want to run Python programs from time to time.

I don't really know Rust, or Cargo, but I never have trouble building any Rust program: "cargo build [--release]" is all I need to know. Easy. Even many C programs are actually quite easy: "./configure", "make", and optionally "make install". "./configure" has a nice "--help". There is a lot to be said about the ugly generated autotools soup, but the UX for people just wanting to build/run it without in-depth knowledge of the system is actually quite decent. cmake is a regression here.

With Python, "pip install" gives me an entire screen full of errors about venv and "externally managed" and whatnot. I don't care. I just want to run it. I don't want a bunch of venvs, I just want to install or run the damn program. I've taken to just use "pip install --break-system-packages", which installs to ~/.local. It works shrug.

Last time I wanted to just run a project with a few small modifications I had a hard time. I ended up just editing ~/.local/lib/python/[...] Again, it worked so whatever.

All of this is really where Python and some other languages/build systems fail. Many people running this are not $language_x programmers or experts, and I don't want to read up on every system I come across. That's not a reasonable demand.

Any system that doesn't allow non-users of that language to use it in simple easy steps needs work. Python's system is one such system.

"I don't want a bunch of venvs"

That's your problem right there.

Virtual environments are the Python ecosystem's solution to the problem of wanting to install different things on the same machine that have different conflicting requirements.

If you refuse to use virtual environments and you install more than one separate Python project you're going to run into conflicting requirements and it's going to suck.

Have you tried pipx? If you're just installing Python tools (and not hacking on them yourself) it's fantastic - it manages separate virtual environments for each of your installations without you having to think about them (or even know what a virtual environment is).

Managing a farm of virtualenvs and mucking about with my PATH doesn't address the user-installable problem at all. And it seems there's a new tool to try every few months that really will fix all problems this time.

And maybe if you're a Python developer working on the code every day that's all brilliant. But most people aren't Python developers, and I just want to try that "Show HN" project or whatnot.

Give me a single command I can run. Always. For any project. And that always works. If you don't have that then your build system needs work.

"Give me a single command I can run. Always. For any project. And that always works."

    pipx install X

Right so; I'll try that next time. Thanks. I just go by the very prominent "pip install X" on every pypi page (as well as "pip install .." in many READMEs).

Yeah, totally understand that - pipx is still pretty poorly known by people who are active in Python development!

A few of my READMEs start like this: https://github.com/simonw/paginate-json?tab=readme-ov-file#i...

    ## Installation

    pip install paginate-json

    Or use pipx (link to pipx site)

    pipx install paginate-json

But I checked and actually most them still don't even mention it. I'll be fixing that in the future.

Out of the 3 things I want to install 2 don't work. Both of these seem bugs in pipx so I reported one, but the feedback was borderline hostile and it ended up being closed with "unclear what you want". I'm not even going to bother reporting the other bug.

So whatever the goals are, it doesn't really work. And in general pipx does not strike me as a serious project.

Pipx is great! Although, I always seem to have to set up PATH, at least on windows?

I could say the exact same stuff about NodeJs, c++, go, rust, php, etc. All of these are easy to use and debug and "install easily" when you know them and use them regularly, and the opposite if you're new. Doubly-so if you personally don't like that language or have some personal pet peeve about it's choices.

Guys let's not pretend like this is somehow unique to python. It's only until about a few years ago that it was incredibly difficult to install and use npm on windows. Arguably the language ecosystem with the most cumulative hipster-dev hours thrown at it, and it still was a horrible "dev experience".

That does not match my experience. I've been working with Python for a year or so and the packaging problems come up every now and then still.

I've installed/built a few packages written in Go and Rust specifically and had no problems.

That single command is pipx.

Pythons venvs are a problem to the solution of solving the dependency problem. Consider the following: it is not possible to relocate venvs. In what universe does this make sense? Consider a C++ or Rust binary that would only run when it is placed in /home/simonw/.

Normal users who just want to run some code shouldn't need to learn why they need a venv or any of its alternatives. Normal users just want to download a package and run some code without having to think about interfering with other packages. Many programming languages package managers give them that UX and you can't blame them for expecting that from Python. The added step of having to think about venvs with Python is not good. It is a non-trivial system that every single Python user is forced to learn, understand, and the continually remember every time they switch from one project to another.

This is correct. The whole application installation process, including the creation of a venv, installing stuff into it, and registering it with some on-PATH launcher should be one command.

BTW pyenv comes relatively close.

I agree with that. Until we solve that larger problem, people need to learn to use virtual environments, or at least learn to install Python tools using pipx.

sudo apt install pipx

pipx install package_name

Takes care of the venv and the script/app path is added to system path.

I reject the virtual environments and have no issues. On an untrusted machine (see e.g. the recent token leak):

  /a/bin/python3 -m pip install foo
  /b/bin/python3 -m pip install bar

The whole venv thing is overblown but a fertile source for blogs and discussions. If C-extensions link to installed libraries in site-packages, of course they should use RPATH.

This is mostly a curse of Python’s popularity. The reason you can’t pip install with system Python is that it can break things, and when your system is relying on Python to run various tools, that can’t be allowed. No one (sane) is building OS-level scripts with Node.

The simplest answer, IMO, is to download the Python source code, build it, and then run make altinstall. It’ll install in parallel with system Python, and you can then alias the new executable path so you no longer have to think about it. Assuming you already have gcc’s tool chain installed, it takes roughly 10-15 minutes to build. Not a big deal.

Its more probable that you are trying to install the deps in the system python. And using pip instal xxxxx -u will install them in your user directory rather than the system. I'm pretty sure modern Ubuntu warns you against doing that now anyway.

If you're installing for a small script then doing python -m venv little_project in you home dir is straightforward, just active it after [1]

I'm using rye[2] now and its very similar to Rust's Cargo, it wraps a bunch of the standard toolchain and manages standalone python versions in the background, so doesn't fall into the trap of linux system python issues.

[1]https://docs.python.org/3/library/venv.html [2]https://rye.astral.sh/

Maybe I am biased, because I learned these things so long ago and I don't realize that it's a pain to learn. But what exactly is so confusing about virtualenvs ?

They really not that different from any other packaging system like JS or Rust. The only difference is instead of relying on your current directory to find the the libraries / binaries (and thus requiring you to wrap binaries call with some wrapper to search in a specific path), they rely on you sourcing an `activate` script. That's really just it.

Create a Virtualenv:

    $ virtualenv myenv

Activate it, now it is added to your $PATH:

    $ . myenv/bin/activate

There really is nothing more in the normal case.

If you don't want to have to remember it, create a global Virtualenv somewhere, source it's activate in your .bashrc, and forget it ever existed.

Only python demands you to source an activation script before doing anything.

Yes, though just to illustrate that it's a matter of taste, I do prefer the solution of virtualenvs requiring to source a script that append to PATH, rather than a solution requiring the use of a wrapper that executes in its PATH.

I never remember how to run Javascript binaries. Is it npm run ? npm run script ? npx ? I always end up running the links in node_modules/bin

Do you have a problem with Node.js too because it creates a node_modules folder, or is the problem that it is not handled automatically?

I don't care about the internals. I care about "just" being able to run it.

I find that most JS projects work fairly well: "npm install" maybe followed by "npm run build" or the like. This isn't enforced by npm and I don't think npm is perfect here, but practical speaking as a non-JS dev just wanting to run some JS projects: it works fairly well for almost all JS projects I've wanted to run in the last five years or so.

A "run_me.py" that would *Just Work™" is fine. I don't overly care what it does internally as long as it's not hugely slow or depends on anything other than "python". Ideally this should be consistent throughout the ecosystem.

To be honest I can't imagine shipping any project intended to be run by users and not have a simple, fool-proof, and low-effort way of running it by anyone of any skill level, which doesn't depend on any real knowledge of the language.

Like people with actual lives to live and useful stuff to do that's not learning about and hand-holding a dozen different half-baked build systems.

But sure, keep up the cynical illusion that everyone is an idiot if that's what you need to go through life.

This is the truth right here. The issues are with people using (not officially) deprecated tools and workflows, plus various half baked scripts that solved some narrow use cases.

All is well, then, one day, you have to update one library.

Some days later, in some woods or cave, people will hear your screams of rage and despair.

Been using python for 15 years now, and these screams were never heard.

Dev/test with relaxed pip installs, freeze deployment dependencies with pip freeze/pip-tools/poetry/whateveryoulike, and what's the problem?

same here. Been using python/pip for 10+ years and this was never a problem. In the java world, there is jar hell, but it was never a crippling issue, but a minor annoyance once a year or so.

In general, is dependency management such a massive problem it is made to be on HN? Maybe people here are doing far more complex/different things than I've done in the past 20 years

Guessing that folks who write such things are lacking sysad skills like manipulating paths, etc.

It does take Python expertise to fix other issues on occasion but they are fixable. Why I think flags like ‘pip —break-system-packages’ are silly. It’s an optimization for non-users over experienced ones.

Deps in CPython are more about .so/.dll problem, not much can be done since stuff happens outside python itself.

The shitshow that is python tooling is one of the reasons I prefer java jobs to python jobs when I can help it. Java got this pretty right years and years and years earlier. Why are python and javascript continuing to horse around playing games?