SmaCC: Re-parsing strings as expressions

[joelreymont]

I'd like to parse this

@define I32     "(OpSz=0)"

### ALU Instruction with Immediate ###

:addi  Rt, Ra, Imm15s is $(I32) & Opc=0b101000 & Rt & Ra & Imm15s { Rt = Ra + Imm15s; }

Here @define I32 "(OpSz=0)" is a preprocessor definition and $(I32) is its use.

"(OpSz=0)" is normally a string. Is it possible to re-parse it as something else if the string in a preprocessor definition?

I'd like to avoid implementing a separate preprocessor if possible.

@j-brant

[j-brant]

Depending on your grammar, you may be able to treat it like a function call. Most likely you won't be able to do that and then the solution is to either implement a preprocessor or do some sort of tricks with how you handle tokens coming from the scanner. For example, you could expand the tokens for the $(I32) into a collection of tokens ( OpSz = 0 ) when you process the $(I32).

[joelreymont]

What is like to do is disable the string token from the method associated with the @define token, then enable it back when done processing. Can I do this and how?

[joelreymont]

This is it exactly! I would like to expand expand I32 into its collection of tokens. I need to capture those tokens first, though, and right now they are captured as a string. So how would I set up this expansion with SmaCC?

[j-brant]

You could do something like override the getNextToken method. If the next token is a macro, then you would look it up and use the tokens it produces for the next calls to getNextToken. If you are using a glr parser, then you'll need to save the tokens into the state and make sure they are properly copied when duplicating the state.

[joelreymont]

John, this is great advice! But how and when would I parse the original macro definition string to get its tokens?

Is there a way to enable/disable tokens at runtime? I think all my problems would be solved by disabling the string token at the beginning of the macro definition production and re-enabling it at the end. There can be no strings within a macro definition so the approach should work.

[j-brant]

Without knowing the specifics of what you are doing, there are a couple ways you can do things. One would be to define a scanner method for the @define token that would register the name with the string, and then when expanding the macro would run the scanner on the string to get the list of tokens. This could all be done in the scanner. Another approach would be to define a @define production in the parser that took the name and list of tokens. Then when you see the macro reference while parsing, you already have the list of tokens to be returned.

[joelreymont]

One would be to define a scanner method for the @define token that would register the name with the string, and then when expanding the macro would run the scanner on the string to get the list of tokens.

Do you have any examples or code pointers for how to do this?

Can I define a @define production in the parser that takes a name and tokens when I don't know in advance what the tokens are? The rule is that an expression is supplied within the string and is used to replace future uses of $(I32), etc.

[j-brant]

You could do something like this:

<define>
	: \@define
	;
<misc>
	: \{ | \} | [a-zA-Z]+
	;
<whitespace>
	: \s+
	;
<semicolon>
	: a
	;
Start 
	: "Foo"
	;

And define a define method on your scanner:

define
	| token |
	[ token := self next.
	token ids first = self emptySymbolTokenId or: [ token value = ';' ] ]
		whileFalse.
	^ self next

With this you can parse @define asdf {{}}{{}{{} ; Foo. It will skip over the @define stuff and only see the Foo token. For your use, you would need to store the tokens so you could replay them when you see the reference.