What's the history behind this fork and what is it likely to mean in terms of fragmentation? In particular what are the irreconcilable beliefs on each side that have led to this?
I haven't tried Scala yet, but my gut reaction is I should stick with Clojure until the dust settles. A fragmented community around a relatively niche language is never good.
As someone who's been using Scala for the better part of a couple years:
This seems more like a more ambitious version of the Shapeless and ScalaZ libraries (with some of the same crowd/authors).
I (and I suspect most people who program in Scala for a living) don't use those libraries. Not because they're in any way bad. But there are trade-offs using them IME. The biggest being compile-time. So I get by fine without.
I think this is probably generally good news. In the same way that (I believe) JRuby pushed MRI forward and popularized the case for a multi-threading VM, more advanced type features in this fork probably benefit the Scala community as a whole.
I certainly don't see how it could be harmful. Typesafe isn't going away. The vast majority of businesses will still use their compiler. If you didn't use the advanced-type-based-libraries before, this news probably isn't going to impact you anyway.
So from where I'm standing, having made a late 2.9 to 2.10 transition, and now having made several 2.10 to 2.11 transitions, Scala seems a very mature language/platform to me. Libraries are generally upgraded fairly quickly, and source wise, those releases broke very little (2.11 being the biggest culprit just because of the modularization effort, but that's pretty trivial to resolve).
I dunno. It's not Java. But it's not Ruby either. It's gotten new features since 2.9, but it's broken or changed very little. I find working with it, even working through upgrades, very stable and mostly frustration free.
As someone involved in the scala community (and scalaz) community, I'd say about 60% of the companies I talk to at the various meetups use at the very least #scalaz. I was pleasantly surprised.
Could this be a biased sample though? Not saying this is the case but it seems logical that the people at the scala meetups are cut from a cloth that would make them more likely to be using things such as scalaz.
I know that my local scala meetup seems to be more heavily oriented towards the functional end of things than the bulk of the scala developers I interact with. OTOH, perhaps the bias is with my sample.
The fork means nothing dramatic. I'll make a short summary of both positions:
Typesafe: we need to offer our clients a good guarantee on stability and compatibility with Java 8 or they won't take Scala seriously. So, for the moment we won't introduce new features or make big changes in the compiler.
Typelevel: we are OK with compatibility and stability, but there are some features that we really want which either don't exist or are a bit quirky with the current compiler. So we will make a fork in which we can add these things while staying binary and merge compatible with yours.
If they make it work, this is great news. There wouldn't be fragmentation as "bleeding edge Scala" would be binary and merge compatible with "stable Scala". And nobody should have hurt feelings, both positions are completely sensible.
Anyone who tries to make the claim that "pretty much hardly anyone uses" is standing on shaky ground. It definitely has not surpassed Java in popularity but it seems to be quickly gaining traction at lots of large companies and is being used more and more. I was recently at Scala by the Bay in San Fransisco and it seems each year more and more companies are on board. If LinkedIn, Twitter, PayPal, Cisco, Verizon, and more are all using Scala I would hardly call it a niche language anymore.
There are a few really big and impactful things already implemented in Scala, and some companies may be using it without even actually writing any Scala code (for they provide Java APIs): Play, Kafka, Spark, Akka.
Yes, let's cite this guy who is "so uninterested in Scala" that he spends an considerable amount of time maintaining multiple user accounts to spread FUD about the langauge in various places and makes up stories about how "his team/group/company/... was trying to use Scala but abandoned it due to all the horrible issues with it".
You know that he is really persistent (despite claiming that he doesn't care about Scala) when the community starts maintaining a list of his sockpuppet accounts.
>>In particular what are the irreconcilable beliefs on each side that have led to this?
I'm not familiar with this one. But in general most of these forks are generally out of personality clashes where one person just refuses to take another as the boss.
Scala is still not among the top languages used in the industry. This kind of my way or the highway attitude will only fragment the already fragile ecosystem. Imagine any serious software shop wanting to use Scala reading this. If I want to be writing an enterprise application which I plan to keep running for the next decade, should I be choosing a language which isn't just not widely adopted, but the existing ecosystem itself has forks because a few members don't give two hoots about stability(Which is far more important to 99% of the software shops out there, than adding new features).
Instead of doing something like this, and putting your broader community of users in perpetual confusion as to which ecosystem to use, you could rather contribute to something meaningful.
I did, and its the same template story always. We don't agree with technical decisions made by Set A people to be in best interests of the community, therefore we plan to fork Y, with intelligent Set B people.
> What's the history behind this fork and what is it likely to mean in terms of fragmentation? In particular what are the irreconcilable beliefs on each side that have led to this?
What I've managed to put together, corrections welcome:
Typesafe has been moving to stabilize Scala, primarily to make it more attractive to enterprise clients. Prior to 2.10, even point releases broke binary compatibility; now compatibility is maintained for those and they're making efforts to make minor releases binary compatible. The tooling they've produced and the testing framework that builds all the major libraries in the ecosystem and checks for binary compatibility means they now have quite possibly the best binary compatibility regime of any language.
Typesafe recently published a roadmap for the next few releases, saying that the next release would focus exclusively on Java 8 support and remain compatible with the current release. This means work to merge any new features won't even start until January 2016, and those features are unlikely to be in an official release until 2017.
Less frequent and more compatible releases are what a lot of people have been asking for, and it would be very awkward to maintain Java 8 and pre-Java 8 releases of Scala. So the typesafe position that there can be no new features in the release that introduces Java 8 (so that people will definitely be able to switch to Java 8 without making any code changes) makes a certain amount of sense. At the same time it's meant Scala's getting "frozen" in a bit of a weird state; macros were a recently-introduced experimental feature and the official version is kind of half-finished (there's a branch with a more complete version). And there are a couple of features that are quite small and simple but would be a tremendous help for the kind of code the typelevel guys work on (e.g. partial application of types would be a very simple syntactic rewrite, and there's already an implementation as a compiler plugin), and there's long been a feeling that they were coming as soon as we figured out the little details - so to then be told that we won't see these features until 2017 at the earliest was a bit upsetting.
So there was a bit of consternation among the typelevel folks - a group who maintain some quite abstract and powerful libraries that tend to push the type system a bit further than most user code. Some of them started seriously talking about porting their libraries to one of the smaller, more experimental languages with a more elegant type system (Idris, Ermine, or I forget the other option). At the same time, the Scala community now has a lot of expertise in maintaining binary compatibility across quite complex changes, so at some point Miles came up with this idea. I don't think a fork would have happened otherwise - most of the goals can be achieved by compiler plugins (at the cost of being harder to maintain), and if they were going to break compatibility they might as well have jumped all the way to another language. But if the typelevel guys can pull off this "conservative fork", where they'll use their own compiler to compile their libraries but those libraries will then be binary compatible with "vanilla" scala and usable by ordinary scala users, then they get the best of both worlds. It's ambitious, but I hope it works out.
> I haven't tried Scala yet, but my gut reaction is I should stick with Clojure until the dust settles. A fragmented community around a relatively niche language is never good.
If they're maintaining binary compatibility then that should keep fragmentation to a minimum. Honestly I don't think this changes much compared to how it was with compiler plugins - if you're building some super-duper-typey project with these advanced features then maybe you'll muck around and use this forked compiler, but if you're a normal user then you can ignore it and carry on using their libraries without noticing any different.
Plus even if the Scala community were to split in two, I think either half would be bigger than the Clojure community.
> Typesafe has been moving to stabilize Scala, primarily to make it more attractive to enterprise clients.
I work primarily for startups and have been using Scala for the last 3 years. Typesafe is entirely in the right to want to deliver a more stable Scala, not just for some mythical enterprise, but for all people using Scala for real-world projects. And this is because people complained about the lack of stability. This isn't just because Typesafe has products to sell, it's also because if you want a language to succeed, then it needs to cater to real world concerns.
Lets compare the situation with Clojure for example - with which you can usually use a library created for Clojure 1.1 and that hasn't been updated in years and that's still usable on top of the latest Clojure version.
Personally I'm happy that this fork happens, because:
1. if it adds any good features, they'll be merged,
and that's what open-source is about
2. if it fails, then there's no harm done
On the other hand I do not have faith in the ability of the Typelevel people to pull this off. Like every time I tried playing with Scalaz, I ended up with stack-overflow errors - not to diminish their skill or efforts, as yes, they are amongst the very best developers, but these people are too idealistic to address the needs of projects that solve real world problems.
Miles himself speaks with contempt about Play and Akka and their Java compatibility - but other project leaders have a lot to learn from these projects, as they provide a blueprint for how to achieve success - a documentation first effort, reasonable stability, gradual learning curve, a sane deprecation process, mindfulness about the platform they run on and so on. Compare with Lift. Or Scalaz.
Ok, time for some personal opinions. I agree that binary compatibility is important, and that Typesafe's recent focus on stability has been correct. I do however think that the planned focus on Java 8 to the exclusion of all else is misguided.
* Scala on Android users are very worried by this direction, because Java 8 doesn't and won't run on Android; Typesafe has not put forward a good narrative about Android. If Typesafe is simply abandoning any support for Android that's... possibly a reasonable commercial decision, but rather upsetting for those of us who were using it. Either way, more clarity would be better.
* Scala users, including users who use Java libraries in Scala, are mostly unconcerned about what bytecode their code compiles to - if it does the same thing either way then it doesn't really matter whether it's Java 7 or Java 8. Performance and code size improvements are always welcome, but not worth freezing the language for 2 years for.
* The only users who really benefit from the Java 8 support are users who want to use Scala libraries in Java (since they want to be able to e.g. pass Java 8 lambdas as callbacks). This is evidently a constituency that's commercially important to Typesafe, since they put a lot of effort into making Play and Akka usable from Java. But I don't think this reflects the wider Scala ecosystem at all; most Scala libraries are written to be used from Scala, with little if any concern for Java support. IMO the primary (not exclusive, but primary) focus of the Scala compiler should be supporting Scala programmers (some of whom desperately want features like partial type application), not providing libraries for Java programmers.
* So who are the users who are actually benefiting from this? Only those who are both cutting-edge enough to be moving to Java 8 (and to use Akka/Play), but conservative enough to not want to use Scala for their code. That seems like a fairly small set to me.
I agree with the value of documentation and deprecation cycles, and that there are Typelevel projects which should make more effort in these directions.
> Like every time I tried playing with Scalaz, I ended up with stack-overflow errors - not to diminish their skill or efforts, as yes, they are amongst the very best developers, but these people are too idealistic to address the needs of projects that solve real world problems.
Huh? That's not a question of idealism, that's just a bug - and "practical" programmers introduce plenty of those. I've never hit a stack overflow in Scalaz FWIW, but sure, sometimes programs have bugs. I've found the typelevel projects to be relatively high-quality code though.
On Java 8 - it's not just because of the purpose of interoperability, but also for the runtime improvements. Things like invokeDynamic, ConstantCallSite and others are not in Java 6. Upgrading the requirement means opening the door to performance improvements and is a real concern of many. How much it can be achieved in Scala I don't know, but these improvements in the JVM have made a dramatic difference for other JVM languages.
Going back to Android - it's not right for Android to hold back the Java ecosystem. I've been rooting for Google in their lawsuit with Oracle, but they should also get their act together and improve Android's runtime and toolchain to keep it up to date with the latest Java.
As this is not going to be a problem just for Scala developers - this is going to be a problem for the whole ecosystem, as Java libraries will eventually have good reasons to upgrade from Java 6 too. For example value classes are coming and possibly generic specialization for those value classes in the next versions (like 9 or 10, so it will happen, it's on the roadmap) and there are libraries that desperately want to avoid boxing. So what then? Will we stay with Java 6?
In fact I think this upgrade to Java 8 is sort of anti-enterprise. How many companies have you see with projects running on anything newer than Java 6? There are still companies running with Java 1.4 for that matter. Java 8 is bleeding edge and it will remain to be bleeding edge in 2016.
> For example value classes are coming and possibly generic specialization for those value classes in the next versions (like 9 or 10, so it will happen, it's on the roadmap) and there are libraries that desperately want to avoid boxing. So what then? Will we stay with Java 6?
Well, Scala already does value specialization even on Java 6, so that's less urgent. You are right of course, but I think most libraries will simply release new versions that require Java 8, rather than having an intermediate version with no changes except a Java 8 port. But I could be wrong.
> In fact I think this upgrade to Java 8 is sort of anti-enterprise. How many companies have you see with projects running on anything newer than Java 6? There are still companies running with Java 1.4 for that matter. Java 8 is bleeding edge and it will remain to be bleeding edge in 2016.
I agree - which makes the Typesafe stance rather perverse. Are there really going to be people in 2016 who want to move their Scala to 2.12 so they can call it from Java 8, but would be put off if 2.12 contained any new features? Who won't have anyone available to fix even the most minor of syntactic changes? (or not even that - Typelevel is talking about a compiler that accepts all valid Typesafe-scala programs, with all the new syntaxes being things that are currently errors). I really don't think including partially applied types, byte/short literals, or singleton types in Scala 2.12 would be a dealbreaker for anyone (though I guess Typesafe has to draw their line in the sand somewhere).
What I expect/hope will happen is that Typesafe will accept pull requests for those pieces and some fixing up of macros, which will be enough to make a Scala I'm happy using until 2017. Then this fork can either become a place for further experimentation like Meta, or quietly die; important libraries will be able to continue to support Typesafe Scala, Typesafe will have accepted the proven features that users want without opening the floodgates to every experimental piece under the sun, and the community can hang together.
Well I have those hopes too - this can be a very positive thing, since it can attract more contributions to Scala.
After all, the Scala core probably has their plate full and could use some help. It's one thing to make a proposal for a feature to a group of developers with their hands full, it's quite another to submit a pull request saying "look, this is useful, people like it" - it changes the equation.
I think the idea behind keeping 2.11/2.12 as similar as possible is to enable people to cross-build things more easily, which would drastically improve the life span of the last Java6-compatible release (2.11) until people have found better solutions.
(I can imagine that there will be generic translators from Java8 bytecode to Java6 bytecode, so why should they keep maintaining two backends, if they could maintain one and just leverage a common tool to do the rest?)
I haven't tried Scala yet, but my gut reaction is I should stick with Clojure until the dust settles. A fragmented community around a relatively niche language is never good.