For me, one of the simplest explanations of the difference between macros and functions is the short-circuiting and.

In many languages, and does not evaluate all of its arguments, only as many as it needs to until it finds #f. This sounds like an easy thing to implement, but in most languages it actually isn't, because order of nested operations means that without being asked, it will evaluate it's arguments before our and.

So we can write this broken-and easily enough:

  (define (broken-and . rest)
    (cond
      [(null? rest) #t]
      [(car rest) (apply broken-and (cdr rest))]
      [else #f]))

But give it the following, and you get an error:

  > (broken-and (zero? 0) (string? "dave") (null? '(is fat)) (/ 5 0))

  . . /: division by zero

And that will work so long as all your args are valid. But what will work better is to write a macro. Macros aren't evaluated, they merely splice in their syntax, but we can write them a lot like code, using all our control structures and so forth, and in Scheme/Racket, also some cool pattern matching stuff):

  (define-syntax my-and.v2
    (syntax-rules ()
      [(_) #t]
      [(_ a) a]
      [(_ a b ...) (if a
                       (my-and.v2 b ...)
                       #f)]))

Now we can try our input from before, and get what we want, because the macro expands recursively and won't ever get to our divide by zero, at least unless all preceding args are true:

  > (my-and.v2 (zero? 0) (string? "dave") (null? '(is fat)) (/ 5 0))
  #f

This property of only expanding, not evaluating, is why macros can write all kinds of clever control structures, because you're mangling syntax, not evaluating code.

Fundamentally it's macros and homoiconicity of code and data that give lisp it's expressive power. Combined they grant you the power to inject new syntax / domain specific languages (DSLs) directly into your code. Any new syntax you create can look like normal lisp or can have different syntax evaluation rules like infix math notation [1]. A number of years ago I did a short write-up on stack overflow giving a basic explanation of how macros work in common lisp which may be worth a read [2].

Macros in general work by being a syntax tree -> syntax tree transformation. The neat trick is it's not a static pattern like C - you can leverage functions, conditions, etc to build that syntax tree.

As for comprehensions themselves, if you want to see the power of lisp look at the loop macro. It's a single macro yet gives you the power of normal control loops, list comprehensions, for-each all rolled into one package. [3]

[1] http://data-sorcery.org/2010/05/14/infix-math/

[2] http://www.gigamonkeys.com/book/loop-for-black-belts.html

[3] http://stackoverflow.com/questions/267862/what-makes-lisp-ma...

Python natively supports this sort of list comprehension (https://docs.python.org/2/tutorial/datastructures.html#list-...), and the Lisp source is being translated into a Python AST, so that snippet is probably being directly mapped into whatever AST form is necessary to invoke the list comprehension.

Yes that's exactly what's happening. Hy has two levels of things mostly: a "compiler" that does the low-level mapping of base python constructs (and whatever can be mapped to AST)... here's where list-comp is implemented: https://github.com/hylang/hy/blob/master/hy/compiler.py#L137...

Anything above that is written as Hy macros, in Hy itself: https://github.com/hylang/hy/blob/master/hy/core/macros.hy

There's a lot of other core code that's written in Hy itself too. https://github.com/hylang/hy/tree/master/hy/core

Yes thats correct. https://github.com/hylang/hy/blob/dcf29d3d2a147fe66a19971214... Source for those interested. Note that Hy replaces all - in names for _, thus we can have prettier names :)

Oh, and earmuffs become EARMUFFS, which is also nice.

You can create a list comprehension using a macro. In common lisp, lisp comprehension is implemented by the loop macro. http://www.gigamonkeys.com/book/loop-for-black-belts.html

It is a little more involved, but basically you define a syntax for list comprehension and then parse it using the macro to generate the final result.

So you sort of write stuff that defines your language as a 'macro' (and it looks like the rest of the lisp?) and it executes on your code first to transform it?

Yes, thats where the power of lisp is. Say we want to implement do/while. In Python you would have to go into the compiler. In a Lisp, like Hy. We can just write it as a macro.

We implement yield-from on Python2 using a macro. http://dustycloud.org/blog/how-hy-backported-yield-from-to-p...

You can create macros that look like functions, but the macro actually has access to the the "program code", so you can do arbitrary transformations.

The implementation is easier if you assume that you have exactly two generators. Here is an example in Racket. It uses define-syntax-rule that is the method to define simple macros (there are more advanced options).

(The program also define string->list/string that splits a list in a list of one-character string, because the result of string->list is a list of chars. You can ignore it safety.)

  #lang racket

  (define (string->list/string x)
    (map (lambda (c) (list->string (list c)))
         (string->list x)))

  (define-syntax-rule (list-comp2 (proc v ...)
                        [u1 gen1]
                        [u2 gen2])
    (map (lambda (u1 u2) (proc v ...))
         gen1
         gen2))
  
  (list-comp2 (list x y)
    [x (range 8)]
    [y (string->list/string "ABCDEFGH")])

  ;==> '((0 "A") (1 "B") (2 "C") (3 "D") (4 "E") (5 "F") (6 "G") (7 "H"))

> but the macro actually has access to the the "program code", so you can do arbitrary transformations.

I like to compare the way lisp works to Javascript and the DOM. Just like a webpage (or its JS code) has access to its internal structure and can do arbitrary transformations, so does lisp except instead of the DOM you have the abstract syntax tree.

There's an excellent paper on how to build list comprehensions (in Common Lisp) on top of simpler iteration constructs:

http://www.ai.sri.com/~latendre/listCompFinal.pdf

I have a (very loose) implementation of this approach at:

https://github.com/TBRSS/cl-lc

Frankly, though, once you've learned the trick, list comprehensions lose a lot of their appeal.

"Real" List Comprehensions in 24 Lines of Lisp[0]

[0]: http://lisp-univ-etc.blogspot.com/2013/01/real-list-comprehe...

Macros (the general Lisp concept) are really interesting. I'm still trying to grok them myself, so if I made any mistakes please downvote and correct away.

A normal Lisp function call might look like this:

(myfunc arg1 arg2 arg3)

where arg1, etc can be literals, variables, expressions. Like in most programming languages, each of the arguments is evaluated and then the function is called with the evaluated values of those arguments. When it's done, a value is returned.

A macro is a lot like a function call:

(mymacro arg1 arg2 arg3)

However, it works differently. First of all, the arguments are passed straight into the macro - if arg1 is something like (+ 1 2), the macro will see (+ 1 2), not 3 (as the function would).

Secondly, the macro is intended to return a list, which can be a function call or another macro call. (The macro could also return a final value.) The list can then be evaluated to either expand another macro, or to call a function to produce a final value. So you can think of a macro as a special sort of function that, rather than taking in values to produce another value, takes in literal code to produce transformed code (which then can be evaluated to produce a final value).

Now, one of the uses of macros that people cite is, for example, implementing your own control flow structures. Why are the two related? Lisps have something which are known as special forms, which look like function calls but don't have their arguments evaluated before being passed in (like how macros work). The reason that matters can be described by a simple example: let's say you wanted to implement a very simple version of 'and' as a function (Clojure syntax):

(defn my-and [left-expr right-expr] (if (not left-expr) false (if (not right-expr) false) true))

However, the problem is that left-expr and right-expr will both be evaluated before my-and can even begin executing, along with the attendant side effects, etc. The problem is that functions can't choose whether or not their arguments should or shouldn't be evaluated - they just get the values of their arguments after they've been evaluated[1]. It's not really possible to implement the traditional short-circuiting 'and' using functions, or many control flow structures where you do not want to evaluate every possible branch unconditionally.

However, macros receive the literal code comprising their arguments, and they can choose to evaluate one, none, some, or all of their arguments depending on their internal logic. This makes them better suited for implementing behavior that resembles those of the Lisp special forms.

[1] This is the difference between call-by-name and call-by-value semantics. Most languages are call-by-value, as described above, but (e.g.) Scala has '=>' and Swift has '@autoclosure' to allow for some sort of call-by-name support.

Rebol has an interesting take on this, in that what Lisp calls "macros" and "functions" are unified in Rebol (as "functions") -- you can specify for each argument whether it is taken as evaluated or unevaluated.

Also related : f-expr, a historical attempt at unifying them http://en.wikipedia.org/wiki/Fexpr .

Abandoned because the near impossibility of compilation when using them ( http://stackoverflow.com/questions/18324743/why-were-fexprs-... ).

And more recently http://web.cs.wpi.edu/~jshutt/kernel.html .

Argument's invalid as Rebol and Red are both members of the lisp family. It is like comparing the similarities of a circle with a ellipse. https://news.ycombinator.com/item?id=8613377

I'm not sure how the argument can be invalid, just because Red and Rebol is a lisp? Sounds like both them and Dylan might be a good inspiration for a "python lisp" (not that that is what hy is doing atm, as far as I can tell). Contrasting a circle with an ellipse can be useful, sometimes ;-)

> Argument's invalid as Rebol and Red are both members of the lisp family.

I was pointing out an approach taken by Rebol that's different from most languages that support macros that might be interesting to people, not making an argument.

For non-lispers (like myself) look at Rust and Sweet.js, both from Mozilla. I've been experimenting with Sweet.js and it's very cool and powerful, and I still have a lot to learn about macros and how/where they can be used.

How to implement list comprehensions in Lisp is an open-ended topic.

For instance, I did it once in a "researchy" way by implementing monads. With the help of CLOS, I was able to define various monads such as the list monad, identity monad, or state transformer monad (the map, join and unit being methods dispatched on the monad type).

Then a generic monadic comprehension macro provides different functionality based on which monad is selected. The list monad gives rise to a list comprehension. The identity monad gives rise to just sequential variable binding. And the state transformer monad ... to something like a pipeline of succcessive state transformations, I think.

http://www.kylheku.com/cgit/lisp-snippets/tree/monads.lisp

If you just want a list comprehension, it's not very difficult. Basically it has to expand to code which (for example) expresses a nested loop that steps every variable over its respective list. The expression being collected by the comprehension is stuck into the middle of this loop, inside a piece of code which collects its value into a list, which is stored in a hidden local variable. (Perhaps two local variables are used, to keep track of the head and tail of the list for the sake of efficiency.)

ANSI Lisp has functional applicators that process multiple lists. For instance if you want to add together values from two lists, you can do this:

  (mapcar '+ '(1 2 3) '(10 20 30)) 

   -> (11 22 33)

If you wanted to form a cross product of the two lists instead, the obvious thing would be to write a mapcar-like function:

  (mapcar-cross '+ '(1 2 3) '(10 20 30))

   -> '(11 21 31 12 22 32 13 23 33)

This function could be used as the target syntax for a comprehension macro so that, say:

  (list-comprehend (+ b a) (a '(1 2 3)) (b '(10 20 30)))

   -> '(11 21 31 12 22 32 13 23 33)

is macro-expanded into

  (mapcar-cross (lambda (a b) (+ b a)) '(1 2 3) '(10 20 30))

Translating (transliterating, really) the list-comprehend syntax into mapcar-cross is not very difficult: it's just some very straightforward nested list manipulation. The macro has to collect the list of variables from the trailing arguments after the expression, to form the (a b) argument list of the lambda. It has to remove the variables from those trailing arguments to produce the list expressions like '(1 2 3) and '(10 20 30) that will be applied to mapcar-cross. Then it has to assemble the mapcar-cross syntax.

Of course, you also have to write mapcar-cross, which is your "run-time support function" for your list-comprehend syntax (usable directly without that syntax, too).