I loved using Parinfer when I was writing ClojureScript, but it also made me question the value of Lisp syntax.
If brackets can be inferred from indentation, doesn't this imply that they're extraneous — that indentation only would be sufficient to write many Lisp structures?
One can do that, but most Lisp programmers value the direct "source code is data" nature of Lisp - using s-expressions.
Alternative notations were originally planned for Lisp when it was designed - some books were published using it, but for actual programming it wasn't used too much - especially since most implementations actually a) didn't support it or b) had s-expressions as a default/internally.
Then several attempts to modernize Lisp with a different syntax mostly failed to get traction or failed big (like the Lisp 2 effort).
Then some efforts failed to provide more than one syntax and also failed to appeal to people (Dylan).
What we saw is a bunch of languages derived from Lisp with different syntax (ML, Logo, ...) and a bunch of Lisps for mathematics with different syntax (Reduce, Macsyma, ...).
But the core Lisp programmers were never willing to give up the s-expression-based syntax. It has some clear advantages when manipulating code, especially in interactive programming.
Thanks for these comments. I didn't realise alternative notations were planned and never took off. (Dylan in particular looks really interesting.)
I can see the value of homoiconicity / code as data from the perspective of someone writing a parser.
I've asked what writing code as s-expressions gets you as the end-user of a language before. People usually say, “macros” or “it's easier to manipulate code”. But macros exist in non-homoiconic languages, like Nim and Elixir. And in my short time using ClojureScript I'm not sure what manipulations Lisp syntax gave me that I don't already have with other languages in a modern text editor.
I'd love to see examples of accomplished Lisp users on YouTube or elsewhere, to see what those code manipulations look like.
It's true that macros exist in languages like Nim and Elixir, even C. They either use something more akin to string templates (expanding a macro fills in the holes) or a DSL for manipulating the AST (something new to learn to write code that writes code).
One reason Lispers find macros so beautiful is the simplicity of writing macros using the exact same list processing functions. The code-is-data thing is reinforced through the fact that the AST is one-to-one with the syntax.
> I didn't realise alternative notations were planned and never took off.
See for example this program from McCarthy, 1960. On top is my Common Lisp translation and below you find the original code. The original had to be hand-translated into s-expression syntax to run it in Lisp - in 1960.
> But macros exist in non-homoiconic languages, like Nim and Elixir.
Lots of languages have some kind of macros: C has textual transformations and Elixir has AST transformations.
Lisp macros are different, since they are not AST transformations but token tree transformations. That means Lisp accepts arbitrary token trees (give that the first item is a macro identifier) and one can write programs to transform them.
Languages which support macros for AST transformations, need to have the code parsed into an AST first. That means the expressions it can transform need to be already parsed into an AST - and the language accepted by the parser limits the expressions - unless one can add syntax changes to the parser, too.
ASTs are special purpose data structures, while s-expressions are general purpose data structures and thus are not limited to expressing programs in a particular syntax and are not limited to express something which looks like a program.
Lisp offers a bunch of macros: macros for transforming Lisp code, macros for transforming symbols, compiler macros for optimizing code, macros for transforming s-expressions, ... Additionally the macro has access to a full Lisp - which means it can do arbitrary computations: primitive transformations, complex language transpiling, full parsing&transformation, doing side-effects in the development environment, running at runtime in interpreters, ...
That means for me as a programmer I can automate all kinds of code manipulation. There are many simple and many complex examples.
Another advantage is that the language core can be reduced to a minimum set of syntactic forms and every other syntax can be built on top of it - and the developer uses the same mechanism as the language core: macros. This makes macros very pervasive - a programmer will use a lot of macros and may also program macros.
One of the disadvantages is that macros also cost us: the code we see can be very different from the code which executes and there is an additional programming paradigm to understand: using syntactic abstractions with macros. One can learn some Lisp without touching macros (see for example SICP, which does not use macros), but in real world programs one might see a lot of macros.
An example for a typical use of macros is CLOS, the Common Lisp Object System. It has a three or four - layer architecture:
layer 0 : meta object layer for CLOS - here CLOS is implemented in itself
layer 1 : object layer for CLOS - here CLOS deals with simple representations of itself
layer 2 : functional layer for CLOS - here have functions operating on CLOS objects
layer 3 : macro layer for the CLOS programmer - here we have macros providing ways to configure CLOS programs
Take for example defining a CLOS class:
0) on layer 0 CLOS has CLOS functionality to represent and create instances of classes and metaclasses, he were can also write extensions to new kinds of metaclasses
1) on layer 1 CLOS classes are objects: which have slots which are objects and which have superclasses which are objects
2) on layer 2 CLOS provides functions to create all that stuff: classes, slot objects, ...
3) on layer 3 CLOS provides the DEFCLASS macro which at macro expansion time - which typically will be triggered by compilation - can do all the transformations and can interface to the development environment at compilation time.
The programmer will typically use layer 3 to define a class using the macro DEFCLASS. For more advanced use one could use the function ENSURE-CLASS
Common Lisp originally in 1984 did not have any syntax for defining classes. The first CLOS implementation was a Lisp program which implemented all the layers above and provided the necessary syntax to define the new language objects: generic functions, methods, classes and some others. So it was mostly a user-level program which added a full complex object system - with only little code for implementation specific stuff.
Thus the DEFCLASS macro could introduce any syntax it wanted as a user level program - since it is not constrained by a fronted parser. So the CLOS developers came up with something that was convenient for them to define classes. This could mean that it does something trivial by providing a different order to define things, make things shorter to write or that it does something more complex by actually checking the provided class configuration and doing some code transformations.
Thank you so much for this thoughtful reply — I enjoyed the code samples and the details on Lisp's macro system, and I can see how its approach gives you more control to invent on top of the language. It's given me more to read and think about.
dylan maintainer (bruce something) had lots of valuable insights about this
I think I keep lisp syntax~ at heart because its nature made me learn and think recursively which is something of a rare value (and I'm not saying this as a cult, it really made me solve problems I couldn't in ways I was confident it would either work or be clear that my perspective was wrong, very very nice feeling). Another example is that I still prefer car/cdr even though they're laughably cryptic .. but they embody the whole culture (and have a tiny symmetry to it).
> If brackets can be inferred from indentation, doesn't this imply that they're extraneous.
Yes; this deduction is valid.
However, can brackets be inferred from indentation.
Let's start with a trivial case:
[indent]15
Is this object 15? (15)? Or ((15))? How do we apply indentation to distinguish them?
Do we count the spaces? Indentation is two-space so that 15 preceded by no extra indentation relative to the current level is just 15, then 15 preceded by two space is (15), and so on?
And is that really going to be readable?
What if there is whitespace?
material material
;; .. vertical break
15 ;; this is (15) but could be 15 if I don't see the preceding material.
There is ambiguity in indentation. If I see this:
(1 2 3
4 5 6
where 4 5 6 is the last line in my edit window, I know that this list is not well-formed: zero or more items must follow, and then a closing parenthesis. If indentation is used, I have no idea: I could be looking at the complete thing.
I think the example I typically go to is to ask how to automate cutting/moving text around the editor. Having had YAML blow up on my team enough in the past few months, I've grown quite weary of any inference schemes.
Gets even worse, because as soon as you get into the indentation game, you open up the debate about how many spaces. Such that now some of our YAML files are using two spaces, and some more. Portability of the code is basically nil. Worse, it will look right, but won't mean what someone meant for it to.
There are other concerns beyond just writing code. One huge advantage of s-expressions is that your code is written using literal data structure notation. This allows you to serialize code, and to trivially manipulate it using macros as any other data. This is one of the main features that sets Lisps apart from other languages.
That's exactly the conclusion the Haskell community came to, and how they ended up with optional meaningful-whitespace. It's genius and I wish all other languages gave you the ability to optionally choose between braces or whitespace. It could settle the whole Python vs C syntax war once and for all!
> It's genius and I wish all other languages gave you the ability to optionally choose between braces or whitespace. It could settle the whole Python vs C syntax war once and for all!
I really like Haskell as a language, but the fact it isn't based on s-expression is the biggest issue I have with it. Whitespace, braces, etc. are very minor quibbles: they're basically debates about language as a UI. The problem is, that same language is also the API we must adhere to when manipulating code programatically, e.g. writing interpreters/compilers/documentation generators/code formatters/linters/renderers to HTML or LaTex or whatever/static analysers/verifiers/model checkers/IDEs/refactoring tools/etc.
I think it's telling that a language so heavily focused on language implementation (DSLs, etc.) effectively has a single usable implementation (GHC), and a mountain of dead/niche implementations ( https://wiki.haskell.org/Implementations ).
Imagine we're given the path to a file containing Haskell code, and we want to transform it in some way (e.g. replacing calls to one function with another function). The most basic thing we need to do is parse it, but my work on code analysis tools over the past few years has taught me that we can't even manage that reliably.
We might naively reach for the GHC API, but that requires that we set a whole bunch of configuration options that we may not know (language extensions, package databases, commandline flags, etc. collectively referred to as "dynflags"); if we get those wrong, our program crashes saying `the impossible happened!`. We can't, in general, figure out what these should be; the only real solution is to invoke our program via Cabal, and read in the needed values by emulating the commandline flags of GHC. This solution is useless if we have a string of code, without any associated Cabal project.
We might instead opt for a standalone library, like haskell-src-exts. The problem is, those libraries typically can't parse Haskell code found "in the wild". Language extensions are one problem, but another major blocker is widespread use of the C preprocessor (an unhygenic, unsafe macro system based on string substition; which would be largely unnecessary if Haskell used an easily manipulated format like s-expressions instead).
Note that even GHC can't re-use its own ASTs: the Template Haskell extension provides its own AST representation, entirely separate to that used by the compiler's frontend!
Whilst there are layers on top of Haskell like "liskell", which accept s-expressions and produce Haskell code, they're solving the opposite problem: it's easy to convert from s-expressions, since they're so trivial to manipulate. The hard problem is being able to do anything useful with the mountain of existing code which isn't in a nice s-expression format, other than compile it with (some versions of) GHC, if invoked with the right options from (some version of) Cabal; or maybe Stack; maybe after running Hpack; or who knows what else!
> Note that even GHC can't re-use its own ASTs: the Template Haskell extension provides its own AST representation, entirely separate to that used by the compiler's frontend!
Isn't that about the expression problem, though? How would changing the syntax help?
s-expressions are already (a trivially deserialised encoding of) an AST representation. If the Haskell language used s-expressions, or if the "frontend" language was designed to desugar into s-expressions via a standalone pre-processor, then this representation would be available to any code that wanted to use it. Different programs, or different parts of the same program, might decide to convert it to their own specialised datatypes (like those of GHC's frontend and Template Haskell), but those would essentially be details specific to that program (either as an implementation detail or an API). It would have no real influence on how others choose to process the language.
If we think of the current GHC implementation in this way, we find that the only de facto representations of Haskell code is `ByteString` (or `Lazy.ByteString` or `String` or `Text` or `Lazy.Text`, of course ;) ). Since these are generally unparseable (e.g. via haskell-src-exts and friends), it's hard to do much with them (hence the tendency to delve into GHC's own representations).
Note that for my own work I ended up abandoning Haskell syntax altogether in favour of GHC Core. I use Cabal and GHC to do the parsing (since that seems to be the only way to handle real world code), and a GHC plugin to dump out Core as s-expressions http://chriswarbo.net/git/ast-plugin
That’s the attitude of Python with curly braces. But with lisp the significance of indented structures is up to the implementation (`if` and `let` are the same “syntax” but are conventionally indented differently).
>that indentation only would be sufficient to write many Lisp structures?
Lisp's syntax is just a textual representation of the internal abstract syntax tree, so you could represent it any number of ways: s-expressions, indentations, even something like json.
I think s-expressions were just the absolute simplest way to represent them with the smallest parser required. It's really really not much.
tl;dr the killer feature of s-expressions is that they're trivial to parse and generate, which makes it easy to read/write some other "front end" instead (i-expressions, sweet expressions, wisp, etc.).
If brackets can be inferred from indentation, doesn't this imply that they're extraneous — that indentation only would be sufficient to write many Lisp structures?