> This is the first news I've seen on HN in weeks that I am genuinely excited about! I have several AVX-2 hobby projects in Common Lisp, and an AVX-512 machine. It's an unexpected surprise to read this morning that this very useful ISA is suddenly unlocked. I'll be trying it out right away.
Nice to hear! :)
It is the basic compiler support, and lots of instructions added. However, you can't use knor, knoq and similar since they require scheduling of k-masks. Not done yet. But you can certainly use some of avx512 instructions and add yourself if you need some that is not available already. Check https://github.com/sbcl/sbcl/blob/master/src/compiler/x86-64....
On implementation level, it is a codegen layer. It uses a system of macros to generate instructions from a database of instructions. The database is specified manually:
At compile-time, they are converted into "VOPs", i.e. intrinsic functions, which are used by the compiler to emit the actual machine instructions.
> can it auto-vectorize or anything like that?
Unfortunately, it can't.
> are these intrinsics you have to explicitly ask for?
Yes, more like higher-level intrinsics. You get quite some automation, but you are requesting manually what you need. More like a DSL, than pure intrinsics. This is how you can use it (as an example):
(defun count-lines-and-words-ascii (sap size ws-init-state)
(declare (type fixnum size)
(type (unsigned-byte 8) ws-init-state)
(type sb-sys:system-area-pointer sap)
(optimize (speed 3) (safety 0)))
(loop
with loop-end of-type fixnum = (logandc2 size 127)
for i of-type fixnum from 0 below loop-end by 128
with 0x0 of-type u8.32 = (u8.32 #x00)
with 0x20 of-type u8.32 = (u8.32 #x20)
with 0x0A of-type u8.32 = (u8.32 #x0A)
with wa of-type u64.4 = (u64.4 0)
with la of-type u64.4 = (u64.4 0)
with ws-prev of-type u8.32 = (u8.32 ws-init-state)
for c1 = (u8.32-sap-ref sap (+ i 0))
for c2 = (u8.32-sap-ref sap (+ i 32))
for c3 = (u8.32-sap-ref sap (+ i 64))
for c4 = (u8.32-sap-ref sap (+ i 96))
do
(flet ((process-chunk (curr prev)
(let* ((ctrl (u8.32-sat- (u8.32- curr 9) 4))
(ws (u8.32-or (u8.32= ctrl 0x0) (u8.32= curr 0x20)))
(ws-shift (u8.32-alignr ws (u8.32-permute128 prev ws #x21) 15))
(wmask (u8.32-andc1 ws ws-shift))
(lmask (u8.32= curr 0x0A)))
(values wmask lmask ws))))
(multiple-value-bind (wm lm prev) (process-chunk c1 ws-prev)
(psetf wa (u64.4+ wa (u8.32-sad wm 0x0))
la (u64.4+ la (u8.32-sad lm 0x0))
ws-prev prev))
(multiple-value-bind (wm lm prev) (process-chunk c2 ws-prev)
(psetf wa (u64.4+ wa (u8.32-sad wm 0x0))
la (u64.4+ la (u8.32-sad lm 0x0))
ws-prev prev))
(multiple-value-bind (wm lm prev) (process-chunk c3 ws-prev)
(psetf wa (u64.4+ wa (u8.32-sad wm 0x0))
la (u64.4+ la (u8.32-sad lm 0x0))
ws-prev prev))
(multiple-value-bind (wm lm prev) (process-chunk c4 ws-prev)
(psetf wa (u64.4+ wa (u8.32-sad wm 0x0))
la (u64.4+ la (u8.32-sad lm 0x0))
ws-prev prev)))
finally
(return
(loop for j from loop-end below size
with words of-type fixnum = (sum-lanes wa)
with lines of-type fixnum = (sum-lanes la)
with prev-ws of-type boolean = (logbitp 31 (u8.32-movemask ws-prev))
with tlines of-type fixnum = 0
with twords of-type fixnum = 0
for byte of-type fixnum = (sb-sys:sap-ref-8 sap j)
for curr-ws of-type boolean = (or (= byte 32) (<= 9 byte 13))
do
(when (= byte 10) (incf tlines))
(when (and (not curr-ws) prev-ws) (incf twords))
(setf prev-ws curr-ws)
finally
(return (values (the fixnum (+ lines tlines))
(the fixnum (+ words twords))
nil))))))
I have read through your pdf and I think you make a best case for Lisp I have read in last few years.
We could just imagine who programming languages would look like if we abandoned the specialized notations like Haskell or C++, of which C++ really starts to show up that such notations are probably a dead-end. Or if B. Eich was allowed to use Scheme as the built-in scripting language for Netscape. Perhaps we would not need XML and/or Json even Yaml as machine interchange and description formats?
Anyway, I am not familiar with Futamura and yaml spec, your texts in the repo, and your project is the first time I see this. I did a webserach and have read through the Wikipedia page on Partial evaluation which talks about Futamura projections. But can you please ELI5-me about your project: is this a yaml parser generator or is this a DSL/PL language parser generator? Can I specify few rules in a language of my choice, say C, and it will generate a yaml parser and test suite in C language? Or does it mean I can specify a parser for a programming language, say C or a DSL, as yaml production rules in language of my choice, say Common Lisp, and it will generate a parser for C or that DSL in Common Lisp? Or do I understand this completely wrongly? :)
How does this project compares to something like tree-sitter? This AST you build, I have just glanced over it thus far, or use under the hood, could that be exposed somehow, or is it already, to the application? For example, could we build a server in Common Lisp, that reads this various language specs, builds an AST and gives us answers on questions like: is this position in a code for a function definition, or in a comment, and similar? In other words, could we use it for a tool that gives information about meta-data from the source code so we could use it to build tools like LSP servers, indentation servers, syntax highlight and such? Just a curious question, forgive me if I misunderstand what this does.
> setf is like referring to a mutable reference in C++ or Java.
Setf is rather a computation of a reference, than a reference.
A reference in Java and C++ is a pure pointer, with some syntactic sugar in java (you skeep */-> to define and dereference it), and few corns of sugar spread on top of it in C++ (can't be null).
Nice to hear! :)
It is the basic compiler support, and lots of instructions added. However, you can't use knor, knoq and similar since they require scheduling of k-masks. Not done yet. But you can certainly use some of avx512 instructions and add yourself if you need some that is not available already. Check https://github.com/sbcl/sbcl/blob/master/src/compiler/x86-64....