[metaas-dev] parse tree manipulation

[David Holroyd]

Hi there,

On Mon, Jul 23, 2007 at 01:43:01PM -0700, Leo Meyerovich wrote:
> I'm trying to prototype some compiler optimizations and linguistic
> extensions to AS3 and feeling the framework out. I'm running into some
> behaviour I didn't quite expect and think it is because I don't understand
> the appropriate way to manipulate tokens :) I would have loved to used the
> metaas statement manipulators etc., but they don't seem to be very fine
> grained (particularly, statements instead of ast's mean no cps etc
> transforms). It's nice enough having all the antlr-overhead done, though :)

I'm not familiar with CPS transforms; what are they good for?  Are you
adding tail-call optimisation to AS3 or something like that?

FYI, I did make a start at exposing expression-level stuff in the
hand-crafted DOM interfaces (there should be some of this in Subversion)
but lately I've been spending most of my hack-time on projects built on
top of metaas, rather than on metaas itself.


> Anyways, maybe somebody has thoughts on how I'm doing the following
> incorrectly:
> 
> 1. dupTree doesn't entirely duplicate a tree:
> 
> For example, let's say I wanted to copy the type annotation from a var_def
> identifier on to a method_def. Using setChildWithToken, I can move one to
> the other, but copying (by using node.dupTree()) throws
> 
> java.lang.IllegalArgumentException: No startToken: IDENTIFIER
>    at
> uk.co.badgersinfoil.metaas.impl.antlr.BasicListUpdateDelegate.replacedChild(
> BasicListUpdateDelegate.java:220)
>    at
> uk.co.badgersinfoil.metaas.impl.antlr.LinkedListTree.setChildWithTokens(
> LinkedListTree.java:134)
> 
> Context:
> 
> //good, not dup'ing the source
> ((LinkedListTree) getterTemplate.getChild(5)).setChildWithTokens(
>                        0, (LinkedListTree) typeSpec.getChild(0));
> //bad, dup'ing the source
> ((LinkedListTree) getterTemplate.getChild(5)).setChildWithTokens(
>                        0, (LinkedListTree) typeSpec.getChild(0).dupTree());

Yep.  Support for this is simply not implemented yet.  LinkedListTree
probably needs to supply its own implementation of dupTree() to
duplicate all the subclass-specific state.  The fiddly bit will be
duplicating the relevant sub-segment of the token-linked-list, and then
assigning the correct start/stop tokens in the duplicate tree nodes.


> 2. Sometimes, setting over a child doesn't work, but removing the child and
> putting in another does.
> 
> For example, when assigning to a var in a function, let's say I want to
> replace the rhs (new eventStream()) of one of the getterTemplate variables
> above:
> 
> .setChildWithTokens throws
> 
> java.lang.IllegalArgumentException: No startToken: +
>    at
> uk.co.badgersinfoil.metaas.impl.antlr.BasicListUpdateDelegate.replacedChild(
> BasicListUpdateDelegate.java:220)
>    at
> uk.co.badgersinfoil.metaas.impl.antlr.LinkedListTree.setChildWithTokens(
> LinkedListTree.java:134)
> 
> //bad
> ((LinkedListTree)getterTemplate.getChild(6).getChild(1).getChild(1)
>                    .getChild(0).getChild(0))
>                    .setChildWithTokens(2,
> (LinkedListTree)varAssign.getChild(0).dupTree());
> 
> but removing the old rhs and then adding the new seems to work:
> 
> //good
> LinkedListTree elist =
>                    (LinkedListTree) getterTemplate.getChild
> (6).getChild(1).getChild(1)
>                        .getChild(0).getChild(0);
>                elist.deleteChild(2);
> elist.addChildWithTokens(2, (LinkedListTree) varAssign.getChild(0));
> 
> Hopefully there's some magic refresh call I should be calling after these
> transforms and I don't have to do any character munging. I'm still stuck on
> copying nodes (problem 1) but randomly permuting code (2) has eventually
> worked so far. Thoughts?

In the cases where the manipulation doesn't fail, does the code look
'sane' when serialised back to text again?  If you're using dupTree() I
would expect it to be quite broken.

All the code which maintains the token-linked-list as the AST is
manipulated is *incredibly* hairy!  I'm not really surprised that it
breaks when you try to work the AST directly :(

metaas only works because it has lots of unit tests :)

So, it would be really helpful if you could send some simplified
unit tests that demonstrate these problems.  I can then work to make the
tests pass.



Thanks for taking the time to dig into these issues, and letting me
know about them!

dave

-- 
http://david.holroyd.me.uk/