Writing a best-effort portable code walker in Common Lisp

One of the powerful features of the Lisp language family is possibility to extend the language using macros.  Some of possible extensions would benefit from a code walker, i.e. a library for processing code that keeps track of the status of different part of code, for their implementation.  But in practice code walking is generally avoided. 

In this paper, we study facilities useful to code walkers provided by ‘‘Common Lisp: the Language’’ (2nd edition) and the Common Lisp standard.  We will show that the features described in the standard are not sufficient to write a fully portable code walker.

One of the problems is related to a powerful but rarely discussed feature.  The macrolet special form allows a macro function to pass information easily to other macro invocations inside the lexical scope of the expansion.

Another problem for code analysis is related to the usage of non-standard special forms in expansions of standard macros. We review the handling of defun by popular free software Common Lisp implementations.
       
We also survey the abilities and limitations of the available code walking and recursive macro expansion libraries.  Some examples of apparently-conforming code that exhibit avoidable limitations of the portable code walking tools are provided.

We present a new attempt to implement a portable best-effort code walker for Common Lisp called Agnostic Lizard.