> People have to realise these are mostly for hobby. It is really hard to get these working with other libraries.

I have been dealing with this type of feedback for almost 20 years at this point: "This library doesn't work for my needs, so the conclusion is that it is hobby/not useful/etc for everyone". In your experience, we are not there yet (and we may never be), but the generalization does not hold.

Take Explorer, it was the first library of the bunch to be used in production because it was created by Chris Grainger to write data pipelines at his company. And the blog post does include the link to a talk on production usage.

Other projects, like Nx, started as an experiment, but nowadays it is used in production quite a bit and we spend a lot of the time refining it based on feedback we receive.

No one is claiming those libraries are complete. It is still pre-1.0. But we claim to be evolving and improving based on feedback. So if anyone wants to try something different than Python, and you can afford to engage, give feedback to maintainers, and even contribute, then you will hopefully have a good time and be closer to running it in production too.

Thank you Jose, for your 2 decades of relentless work. I have not disparaged any maintainers or their work. I get it from your comments that these projects are in early days of adoption and development.

From my perspective, just because Chris Grainger does data pipelines in production does not prove the project is for everyone. It means it worked out for him.

May be seeing more updates, papers, interviews, podcasts about their usage in production will change things.

For now, my assessment is everything is wonderful till one hits a limitation of these libraries. If it works out, its wonderful, if not its like neither here nor there.

That's precisely my point. Because Chris runs it in production, it doesn't mean the project is production-ready for everyone. And the fact you ran into roadblocks, doesn't mean they are "mostly for hobby" or are "half-baked". Your initial comment lacked this nuance and rushed into generalizations, which I did my best to clarify.

I’ve been considering using elixir for a while. With these libraries, it’s now entirely feasible to move my backend over to elixir, getting rid of 95% of my python code in the process. The tasks leftover that still need python can easily each fit in their own <100 line file and work just as well using other languages to process the data. Most of it needs to be run on a gpu anyways so it’s already sandboxed wrt deployments

It’s not an all or nothing game, and I rue the day that I decided to do everything in python just because I needed a couple LoC to call a python module.

At this point is isnt trying to convert a happy python user like you. Rather to give tools to teams whose app is already in Elixir and the devs knows Elixir. Instead of bringing in Python to the mix you can use Elixir

Elixir dev you are describing and we use Python for ML stuff because it’s easier to work with the more documented frameworks. These libraries just aren’t “there” yet, and I think it’s important to be honest about that. I’ve tried extending/converting to them, and it just wasn’t worth it yet.

> These libraries just aren’t “there” yet, and I think it’s important to be honest about that.

Exactly. These projects are still during early adoption stage.

While it is production-ready and many are doing so, getting there requires improving documentation, discussing roadmaps with maintainers about a feature you may need (perhaps even contributing it), giving presentations (at your company or at events), and so forth. This can be exciting to some, but a distraction for others. There is certainly plenty of work ahead.

To be fair to the creators of “half baked” libraries, you’re comparing our work to frameworks with several years head start with the backing of trillion dollar organizations (excluding polars). To have convinced ANYBODY to successfully make the switch is, to me, both somewhat of a miracle and an indication that there is a legitimate market for this work. I think it also dispels the hobby claim

As for your issues with Explorer, perhaps it is not the library, but your knowledge of Elixir that was a little too half baked? ;)

Telling someone they’re ignorant probably doesn’t get them to use your tech. Not the person you’re replying to; I use elixir everyday at work.

The biggest barrier to adoption of these libraries I can see is that there is not a wealth of resources to solve the actual problem the person is experiencing.

Googling “dyplr verbs in python” there’s a towards data science post that’s the 6th result. It has step by step code examples to get you started with what I guess the person is looking for.

Googling “dyplr verbs in elixir”, you get the explorer library’s GitHub page and docs as results. I see there’s an example on the first page, but docs present as a high barrier to entry. For better or worse, hand-holding helps get people using the thing, and there’s a SEO-fueled ton of python hand-holding resources out there.

I’m sure you’re aware of these things as the author of the book mentioned, but buying a book is a step for someone who’s already sold on learning the language/framework. Having google-able content is much more useful to get people in the door and in the day-to-day work looking something up.

Until something “cool” and well-known is made with Nx/Axon and friends, they’re gonna be seen as second-class to most people who aren’t elixir diehards. Somehow everyone in elixir-land recs these, but never has anything they actually use them for.

There’s also the switching cost. Most Python ML compute is actually done in C/off-loaded to gpu, so its speed is ok and not greatly constrained by the language choice. Python does have multiprocessing stuff which, while annoying in my experience, makes parallelizing stuff possible. Again, thinking from an outsider’s perspective, there doesn’t seem to be a “must have” reason to use elixir when they’ve already been working in python.

I appreciate your response. I should not have replied in the manner that I did. I think there are many ways to make the argument against using these libraries, and all of the maintainers would agree it doesn't make sense in many situations. I am fine with these arguments, but I don't think it warrants referring to the work we've done as half-baked or hobby projects. Even if not the intention, I think it implies a lack of effort and professionalism which comes across as more disparaging than simply pointing out the gaps.

Sean Moriarity, please use your elixir knowledge to write a user defined function or a plugin in explorer. It's easy to attack other people and assume they are ignorant.

It has always been possible to do custom transformations via Series.transform. And here is a PR adding it to dataframes: https://github.com/elixir-explorer/explorer/pull/912 - the PR uses existing APIs, it was possible before, it just makes it more convenient. I am sure more will be added later

I don't think a lot of people are using these in production yet, but you can't become fully baked without spending some time being half baked.

Sometimes though it's nice to have most of your stuff in one language.

Explorer is still pre-1.0.

The downvotes are for the attitude. If you expected all alternative software options to be strictly better than their replacements we would never get anywhere in terms of DX. Someone has to be brave and be an early adopted and put in the sweat to get shit to work and this guy is just taking potshots instead of making pull requests.

If you suggest what they're correct in and how the replies support your theory, I think the downvotes would be rectified and everyone involved would leave with a better understanding of the topic.

That’s ok, truth is bitter. People may fall for hype sometimes. Maintainers of the project can refute what I am saying but they won’t as this the truth.

- In current explorer one cannot write user defined functions, plugins. - They don’t even have it in roadmap. - Also the exposed functionality is always lagging when compared to polars.

- elixir, nx, explorer,distributed, etc nice keywords - reality is it is still early for them. Better to stay away for production apps in case of ai/ml unless people are connecting to OpenAI and writing chat apps for which they won’t need any of the above except elixir.