It's hard for me to believe that it's easier (and not only more comfortable for the engineers) to start pyston rather then rewrite their backend in go or whatever. Is dropbox really so big (not in sales, in the size of their backend) that they should manage their own python implementation? Also choosing python for their backend really seems like the wrong language.
Compilers (especially JIT-compilers) are hard to get right and even harder to get performant. Since i am interested in these areas i would evaluate Graal/truffle or Jython. I only looked briefly at it, but Jython seems very un-optimised and the JVM really improved it scripting-language support in the recent years. Is there really no way to solve the native-extension problem of Jython?
There is a Truffle implementation of Python, but I think it's a bit out of date https://bitbucket.org/ssllab/zippy. I can't find any papers now but I think they had results showing that it was faster than PyPy (which is consistent with what other Truffle languages have achieved relative to RPython versions).
The native extension problem can be solved in Truffle with the LLVM bitcode interpreter. We're using this to interpret Ruby's C extensions.
thank you. What's the status of truffle? I often hear news from the project, but i never read anything about the scope of the project. Is is highly experimental? Are people using it in production? Is it intended to be used in production in the near future?
Also, if it's faster and the C-extension solved by the LLVM interpreter, what's holding it back from being used?
Truffle is a large active project working on multiple language fronts at the same time (JS, Ruby, R, C). There's a pretty long tail of things to do to go from research prototype to something people could use, but that is what we are working on at the moment. Twitter is experimenting with Graal as a Java compiler in production.
The C extension problem is solved in theory. We're now working on making it something a bit more substantial that the research implementation we have talked about previously.
quick question - the big focus of truffle seems to be js, ruby,R.
Even the github org does not have python - https://github.com/graalvm
your comment today was the first time I realized there was a Truffle port for Python, and is NOT maintained by the parent oracle org.
https://github.com/securesystemslab/zippy
By way of comparison, Carakan (Opera's last JS VM) was under 10 man-years from start to shipping, including all testing resources, etc. (And yes, I realise man-years is a terrible way to judge this!)
How much of Dropbox could they rewrite for something quicker than CPython in the time taken to write a JIT? That's the ultimate business question, and there's plenty of evidence to suggest that in general JITs have been comparatively easy versus a rewrite. As the article says, they have millions of lines of code.
At the same time, at least from my knowledge of the languages, JS, Lua, and PHP are all easier to optimize than Python is (because you can do crazier stuff with Python's object model). JS and Lua also benefit from their incredibly small standard libraries which also means you don't need to potentially spend lots of time reimplementing libraries; CPython at least has most of its being implemented either in Python or through its C API which means if you have C API and licensing allows you can just pull out much of it from there.
It's hard for me to believe that it's easier (and not only more comfortable for the engineers) to start pyston rather then rewrite their backend in go or whatever.
You are correct, it is easier to rewrite in Go. That's what Dropbox learned after putting two FTEs on Pyston for a few years.
But the larger Python community outside of just Dropbox would have really benefited from these two guys continuing to work on Pyston.
Of course, getting a fast JIT-Compiler for Python would be awesome! I have been doing some python lately and it's a nice language. I would guess there is such a broad usage in the industry that a committee funded by various companies could take ownership of the project and greatly increase the assigned resources.
I just question whether it's the right move for dropbox. Seems like not actually tackling the problem and instead trying everything so that it just disappears.
Facebooks Hack could be an interesting comparison here.
Afaik it started out as experiment, driven by one or two engineers and got to a significant usability level before it got more resources.
Even if you have 10 or 30 engineers working on it, if they can speed up the other code written by thousands of developers, it can still materialize a huge gain.
Compilers (especially JIT-compilers) are hard to get right and even harder to get performant. Since i am interested in these areas i would evaluate Graal/truffle or Jython. I only looked briefly at it, but Jython seems very un-optimised and the JVM really improved it scripting-language support in the recent years. Is there really no way to solve the native-extension problem of Jython?
But i maybe wrong, any comments?