>From email@example.com Thu Jan 4 13:24:32 1996
>To: firstname.lastname@example.org (Paul Wilson)
>Subject: Re: Perl, English, syntax for Scheme and shells (was Re: scsh in scm
>Paul Wilson wrote:
>> Instead of macroexpanding as a conceptually
>> separate pass, it's integrated right into the compiler in a clean way.
>The "scope resolution work" being done by the "compiler" is typically
>done in the parsing phase.
>Therefore, you seem to be saying that it's
>better to integrate macro-expansion and parsing. But we already know
>that; it's a myth that the two can, in fact, be kept separate.
>If you look at "the" Scheme macro system (as per the R4 appendix or
>related documents), you will find that it is _impossible_ to perform
>macro expansion as a pure pre-processing phase without embedding a
>good deal of the parser into the expander. This is independent of the
>presence of hygiene.
I'm not sure what you mean here. Standard Lisp macros do exactly
this---with all the consequences we've come to know and hate. There's
a purely surface syntax that macros understand, and it's up to the programmer
to ensure that none of the transformations have unintended scope
effects or otherwise make further parsing come out wrong later. Hygeine is
about fixing the scope problem, and requiring more parsing smarts during
My point was just that traditional implementations of hygeinic macroexpansion
actually replicate a lot of the compiler's work in the macroexpander.
They are essentially precompilers, not just macro preprocessors. They have
to compile Scheme with macros into scheme without macros, by a process
far more sophisticated than conventional macroexpansion. They at least
have to know the core syntax and scope rules of the basic language.
This has advantages---you can write a portable macroexpander that works
with dumb implementations of Scheme.
One advantage of doing it in the compiler proper is that you can use it
to control more things about how things are compiled, because you can
*ask* the compiler what it knows at that point in parsing. Eventually
we want to have a superset of hygeinic macros that is really a framework
for reprogramming the compiler from inside the language. If you're
willing to use our nonportable extensions, you should be able to write
a macro that queries the compiler's intermediate data structures and
gets information that can be used to control compilation of subexpressions.
We even want to allow macros to annotate the compiler's data structures
so that other macros can use the information. Of course, this is a
research topic. It's easy to provide a wildly extensible compiler,
but harder to set up a comprehensible framework so that people know
how to extend one thing without breaking another.
Eventually we want a layered system that allows you to reprogram the
compiler with varying degrees of sophistication and difficulty, depending
on how hard it is to do what you want to do given the normal way the
compiler works. We want to have a well-defined metaobject protocol
that can reify the most important issues in the compiler without
horribly confusing people who only want to know enough to do simple
For example, it appears easy to implement encapsulated objects (with private
slots) using macros, a little harder to inline accessors of those slots,
harder still to do nifty splitting and customization to make them really
fast. Likewise, it's easy to implement very simple closure analysis
outside the base-language compiler, but harder to do better analyses.
But I'm getting off the topic.
One of the things I'm trying to scope out is how clean a front end
we can have, for languages with superficially very different syntax,
where grammar modifications can be inferred by the macro system and
automatically handled correctly in most cases. This requires characterizing
potentially interacting grammar features, and having a grammar schema
to say what's safe to do. For the hard cases (like backward compatibility
with somebody's crufty old infix syntax) you may have to restrict the
feature set of the language, or hand-code little things that resolve
ambiguities the system can't figure out by itself.
I'm hoping that for an interesting class of syntaxes and language designs,
it'll actually be easier to build a system our way, rather than using
the usual techniques (e.g., a yacc-generated or hand-coded parser...).