antlr3-interop - Man Page

The LEXER macro returns a pointer to the base lexer object, which is of type pANTLR3_LEXER. This is not the pointer to your generated lexer, which is supplied by the CTX macro, but to the common implementation of a lexer interface, which is supplied to all generated lexers.

Provides a pointer to the lexer shared state structure, which is where the tokens for a rule are constructed and the status elements of the lexer are kept. This pointer is of type #pANTLR3_RECOGNIZER_SHARED_STATE.In general you should only access elements of this structure if there is not already another MACRO or standard $xxxx antlr reference that refers to it.

The LA macro returns the character at index n from the current input stream index. The return type is ANTLR3_UINT32. Hence LA(1) returns the character at the current input position (the character that will be consumed next), LA(-1) returns the character that has just been consumed and so on. The LA(n) macro is useful for constructing semantic predicates in lexer rules. The reference LA(0) is undefined and will cause an error in your lexer.

The GETCHARINDEX macro returns the index of the current character position as a 0 based offset from the start of the input stream. It returns a value type of ANTLR3_UINT32.

The GETLINE macro returns the line number of current character (LA(1) in the input stream. It returns a value type of ANTLR3_UINT32. Note that the line number is incremented automatically by an input stream when it sees the input character '

The GETTEXT macro returns the text currently matched by the lexer rule. In general you should use the generic $text reference in ANTLR to retrieve this. The return type is a reference type of pANTLR3_STRING which allows you to manipulate the text you have retrieved (NB this does not change the input stream only the text you copy from the input stream when you use this MACRO or $text).

The reference $text->chars or GETTEXT()->chars will reference a pointer to the '\0' terminated character string that the ANTLR3 pANTLR3_STRING represents. String space is allocated automatically as well as the structure that holds the string. The pANTLR3_STRING_FACTORY associated with the lexer handles this and when you close the lexer, it will automatically free any space allocated for strings and their structures.

The GETCHARPOSITIONINLINE returns the zero based offset of character LA(1) from the start of the current input line. See the macro GETLINE for details on what the line number means.

The macro EMIT causes the text range currently matched to the lexer rule to be emitted immediately as the token for the rule. Subsequent text is matched but ignored. The type used for the the token is the name of the lexer rule or, if you have change this by using $type = XXX;, the type XXX is used.

The macro EMITNEW causes the supplied token reference t to be used as the token emitted by the rule. The parameter t must be of type pANTLR3_COMMON_TOKEN.

The INDEX macro returns the current input position according to the input stream. It is not guaranteed to be the character offset in the input stream but is instead used as a value for marking and rewinding to specific points in the input stream. Use the macro Getcharindex() to find out the position of the LA(1) in the input stream.

The PUSHSTREAM macro, in conjunction with the POPSTREAM macro (called internally in the runtime usually) can be used to stack many input streams to the lexer, and implement constructs such as the C pre-processor #include directive.

An input stream that is pushed on to the stack becomes the current input stream for the lexer and the state of the previous stream is automatically saved. The input stream will be automatically popped from the stack when it is exhausted by the lexer. You may use the macro POPSTREAM to return to the previous input stream prior to exhausting the currently stacked input stream.

Here is an example of using the macro in a lexer to implement the C #include pre-processor directive:

fragment
STRING_GUTS :   (~('\\'|'"') )* ;

LINE_COMMAND 
: '#' (' ' | '\t')*
    (
        'include' (' ' | '\t')+ '"' file = STRING_GUTS '"' (' ' | '\t')* '\r'? '\n'
        {
            pANTLR3_STRING      fName;
            pANTLR3_INPUT_STREAM    in;

            // Create an initial string, then take a substring
            // We can do this by messing with the start and end
            // pointers of tokens and so on. This shows a reasonable way to
            // manipulate strings.
            //
            fName = $file.text;
            printf("Including file '\%s'\n", fName->chars);

            // Create a new input stream and take advantage of built in stream stacking
            // in C target runtime.
            //
            in = antlr38BitFileStreamNew(fName->chars);
            PUSHSTREAM(in);

            // Note that the input stream is not closed when it EOFs, I don't bother
            // to do it here, but it is up to you to track streams created like this
            // and destroy them when the whole parse session is complete. Remember that you
            // don't want to do this until all tokens have been manipulated all the way through 
            // your tree parsers etc as the token does not store the text it just refers
            // back to the input stream and trying to get the text for it will abort if you
            // close the input stream too early.
            //

        }
            | (('0'..'9')=>('0'..'9'))+ ~('\n'|'\r')* '\r'? '\n'
        )
     {$channel=HIDDEN;}
    ;

Assuming that you have stacked an input stream using the PUSHSTREAM macro, you can remove it from the stream stack and revert to the previous input stream. You should be careful to pop the stream at an appropriate point in your lexer action, so you do not match characters from one stream with those from another in the same rule (unless this is what you want to do)

A token manufactured by the lexer does not actually physically store the text from the input stream to which it matches. The token string is instead created only if you ask for the text. However if you wish to change the text that the token represents you can use this macro to set it explicitly. Note that this does not change the input stream text but associates the supplied pANTLR3_STRING with the token. This string is then returned when parser and tree parser reference the tokens via the $xxx.text reference.

While you can create your own custom token class and have the lexer deal with this, this is a lot of work compared to the trivial inheritance that can be achieved in the Java target. In many cases though, all that is needed is the addition of a few data items such as an integer or a pointer. Rather than require C programmers to create complicated structures just to add a few data items, the C target provides a few custom fields in the standard token, which will fulfil the needs of most lexers and parsers.

The token fields user1, user2, and user3 are all value types of #ANTLR_UINT32. In the parser you can reference these fields directly from the token: x=TOKNAME { $x->user1 ... but when you are building the token in the lexer, you must assign to the fields using the macros USER1, USER2, or USER3. As in:

LEXTOK: 'AAAAA' { USER1 = 99; } ;

The PARSER macro returns a pointer to the base parser or tree parser object, which is of type pANTLR3_PARSER or pANTLR3_TREE_PARSER . This is not the pointer to your generated parser, which is supplied by the CTX macro, but to the common implementation of a parser or tree parser interface, which is supplied to all generated parsers.

When used in the parser, the INDEX macro returns the position of the current token ( LT(1) ) in the input token stream. It can be used for MARK and REWIND operations.

In the parser, the macro LT(n) returns the pANTLR3_COMMON_TOKEN at offset n from the current token stream input position. The macro LA(n) returns the token type of the token at position n. The value n cannot be zero, and such a reference will return NULL and possibly cause an error. LA(1) is the token that is about to be recognized and LA(-1) is the token that has just been recognized. Values of n that exceed the limits of the token stream boundaries will return NULL.

Returns the shared state pointer of type pANTLR3_RECOGNIZER_SHARED_STATE. This is not generally useful to the grammar programmer as the useful elements have generic $xxx references built in to ANTLR.

When building an AST via a parser, the work of constructing and manipulating trees is done by a supplied adaptor class. The default class is usually fine for most tree operations but if you wish to build your own specialized linked/tree structure, then you may need to reference the adaptor you supply directly. The ADAPTOR macro returns the reference to the tree adaptor which is always of type pANTLR3_BASE_TREE_ADAPTOR, even if it is your custom adapter.

Returns a reference type of #pANTRL3_BASE_RECOGNIZER, which is the base functionality supplied to all recognizers, whether lexers, parsers or tree parsers. You can override methods in this interface by installing your own function pointers (once you know what you are doing).

Returns a reference to the input stream of the appropriate type for the recognizer. In a lexer this macro returns a reference type of pANTLR3_INPUT_STREAM, in a parser this is type pANTLR3_TOKEN_STREAM and in a tree parser this is type pANTLR3_COMMON_TREE_NODE_STREAM. You can of course provide your own implementations of any of these interfaces.

This macro will cause the input stream for the current recognizer to be marked with a checkpoint. It will return a value type of ANTLR3_MARKER which you can use as the parameter to a REWIND macro to return to the marked point in the input.

If you know you will only ever rewind to the last MARK, then you can ignore the return value of this macro and just use the REWINDLAST macro to return to the last MARK that was set in the input stream.

Rewinds the appropriate input stream back to the marked checkpoint returned from a prior MARK macro call and supplied as the parameter m to the REWIND(m) macro.

Rewinds the current input stream (character, tokens, tree nodes) back to the last checkpoint marker created by a MARK macro call. Fails silently if there was no prior MARK call.

Causes the input stream to position itself directly at offset n in the stream. Works for all input stream types, both lexer, parser and tree parser.