Note: Not at all an emacs expert, but know some stuff about trying to parallelise a long-standing language ( GAP -- www.gap-system.org , 30 year old maths language).
I'd strongly recommend something like Javascript's "web workers", where each thread lives in an almost entirely independent memory space, with sharing limited to explicitly named buffers.
The problem with traditional "everything shared by default" threading is even when you get the basic core working, we found throughout the system there were buffers and caches, which all needed making thread-safe. Even after that, every library needed some fixes -- and while these were often simple these libraries may not have had updates in years.
In principle I agree whole heartedly. When introducing parallelism and or concurrency into an old language, the "web workers" aproach, or Erlangs actors, to name a decades older reference to a similar idea, should always be considered. The problem with emacs in particular is the use of the buffer as the most versatile data structure. Lot's of data that could live in some data structure lives in some overlay over some part of the buffer. And emacs is built on dynamic variables. Others in this thread have argued to "just refactor" them to lexically scoped variables. But that would decimate the largest vector to adapt and modify the system.
Parallelizing Emacs needs to be a long-term endaveour, as the maintainers of Emacs already said years ago. I think Emacs could learn something from Clojure's story for parallelism. Make data structures immutable by default, and reference them from as few global variables as possible. The "make data structures immutable by default" ship has already sailed, but introducing additional immutable list/vector/map/set data structures with first class syntax support would allow newly written code to prepare for a more parallel future. Such immutable data structures could then be shared between different "ELisp web workers". The "as few global variables story" ship has already sailed, but again ELisp authors would start to consider reducing the number of global variables if it would enable multi-threading.
Yeah, I think that's the way... Dart does this well with Isolates[1]. The language is basically single-threaded but you can start Isolates that run on different Threads but have their own independent memory, and they can exchange messages, much like in the Actor model. Not having to worry about real shared memory between Threads as in C++ or Java is incredibly liberating. As others point out, however, for extremely high performance that's not the best solution, but for nearly all applications, including emacs, the Isolate approach is plenty good enough.
I'd argue that emacs already has that with process buffers using sentinels. :D
By and large, if you don't write blocking code, it doesn't block. Big caveat here is the TRAMP logic to establish connections can block. Not sure why those weren't done in a more async way.
one really nasty conflating issue is that emacs uses a mess of globals. partially because of the domain but largely because of dynamic binding and the lack of namespaces.
emacs really needs to be hoisted onto another language and runtime. everyone says this but no one can really conceive of doing this and rewriting or at least refactoring all the elisp extensions.
I guess guile emacs is still alive? I should really try installing it
Guile-Emacs is no longer alive. The only other project similar since then has been remacs, but that aimed to replace the C core, not elisp. It also lost momentum. I believe adding parallelism to elisp will be more feasible and successful than another rewrite. It's not a bad language really.
I wonder if it would be possible to automatically rewrite the elisp code to whatever new runtime was chosen? Assuming the new language is still more-or-less a Lisp ... maybe?
The issue is always the parts where the two language semantics differ too much. And then, you usually have too much legacy code that would be impacted by changes in those areas. That's the hard part.
You lost when that code lies beyond some organizational boundary, e.g., a separate project that has is too big in its own right to refactor in one go, or where the maintainers have sufficient clout to resist such a change. Such a transformation would effectively entail a fork of that project.
WebWorkers are not very useful as a result. The communication overhead of copying data between two memory spaces usually dwafs any apeedups you could get from doing the work in parallel.
It's easier to implement as a language and runtime, but to get truly high performance on today's many thread many core machines you need shar d memory concurrency more like Java
Arrays of bytes are practically useless in Javascript. And the message passing system for NodeJS workers is almost as useless because it's just as alien to the rest of the javascript normal people write as shared byte arrays is. What we needed was message passing implemented as an async operation that awaited a response. Bidirectional emit() operations don't scale.
If you were building something on top of them using wasm then you probably wouldn't even know what the primitive was. But we're not. We're writing Javascript, and it's a damned shame.
When someone takes a really long time to make a really bad solution, there's frequently not enough group will to correct the issue. This will be the bullshit we are stuck with for the next ten years.
True, they are of limited usefulness, but in Emacs you could (probably) make them more useful over time by exposing more and more sharing -- but (I'm again guessing, don't know Emac's insides) it might be easier/safer to start sharing nothing, then work on sharing as much as possible, rather than start with full sharing and then try to make everything work.
I'd strongly recommend something like Javascript's "web workers", where each thread lives in an almost entirely independent memory space, with sharing limited to explicitly named buffers.
The problem with traditional "everything shared by default" threading is even when you get the basic core working, we found throughout the system there were buffers and caches, which all needed making thread-safe. Even after that, every library needed some fixes -- and while these were often simple these libraries may not have had updates in years.