ANTLR3_COMMON_TOKEN_STREAM_struct - Man Page

ANTLR3_UINT32 channel
pANTLR3_LIST channelOverrides
Override map of tokens.
ANTLR3_BOOLEAN discardOffChannel
If this flag is set to ANTLR3_TRUE, then tokens that the stream sees that are not in the channel that this stream is tuned to, are not tracked in the tokens table.
void(* discardOffChannelToks )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_BOOLEAN discard)
Signal to discard off channel tokens from here on in.
pANTLR3_LIST discardSet
Discared set.
void(* discardTokenType )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 ttype)
Add a particular token type to the discard set.
void(* free )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)
Function that knows how to free an ANTLR3_COMMON_TOKEN_STREAM.
pANTLR3_LIST(* getTokenRange )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)
Function that returns all the tokens between a start and a stop index.
pANTLR3_VECTOR(* getTokens )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)
Function that returns a pointer to the ANTLR3_LIST of all tokens in the stream (this causes the buffer to fill if we have not get any yet)
pANTLR3_LIST(* getTokensList )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_LIST list)
Function that returns all the tokens indicated by being a member of the supplied List.
pANTLR3_LIST(* getTokensSet )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_BITSET types)
Function that returns all the tokens indicated by the specified bitset, within a range of tokens.
pANTLR3_LIST(* getTokensType )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, ANTLR3_UINT32 type)
Function that returns all tokens of a certain type within a range.
ANTLR3_INT32 p
The index into the tokens list of the current token (the next one that will be consumed.
void(* reset )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)
Function that resets the token stream so that it can be reused, but but that does not free up any resources, such as the token factory the factory pool and so on.
void(* setTokenTypeChannel )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 ttype, ANTLR3_UINT32 channel)
A simple filter mechanism whereby you can tell this token stream to force all tokens of type ttype to be on channel.
void * super
Whatever is supplying the COMMON_TOKEN_STREAM needs a pointer to itself so that this can be accessed by any of the API functions which it implements.
pANTLR3_VECTOR tokens
Records every single token pulled from the source indexed by the token index.
pANTLR3_TOKEN_STREAM tstream
The ANTLR3_TOKEN_STREAM interface implementation, which also includes the intstream implementation.

Referenced by antlr3CommonTokenStreamSourceNew(), fillBuffer(), reset(), skipOffTokenChannels(), and skipOffTokenChannelsReverse().

Override map of tokens. If a token type has an entry in here, then the pointer in the table points to an int, being the override channel number that should always be used for this token type.

Referenced by antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), fillBuffer(), reset(), and setTokenTypeChannel().

If this flag is set to ANTLR3_TRUE, then tokens that the stream sees that are not in the channel that this stream is tuned to, are not tracked in the tokens table. When set to false, ALL tokens are added to the tracking.

Referenced by antlr3CommonTokenStreamSourceNew(), discardOffChannel(), fillBuffer(), and reset().

Signal to discard off channel tokens from here on in.

Referenced by antlr3CommonTokenStreamNew().

Discared set. If a token has an entry in this table, then it is thrown away (data pointer is always NULL).

Referenced by antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), discardTokenType(), fillBuffer(), and reset().

Add a particular token type to the discard set. If a token is found to belong to this set, then it is skipped/thrown away

Referenced by antlr3CommonTokenStreamNew().

Function that knows how to free an ANTLR3_COMMON_TOKEN_STREAM.

Referenced by antlr3CommonTokenStreamSourceNew().

Function that returns all the tokens between a start and a stop index. TODO: This is a new list (Ack! Maybe this is a reason to have factories for LISTS and HASHTABLES etc :-( come back to this)

Referenced by antlr3CommonTokenStreamNew().

Function that returns a pointer to the ANTLR3_LIST of all tokens in the stream (this causes the buffer to fill if we have not get any yet)

Referenced by antlr3CommonTokenStreamNew().

Function that returns all the tokens indicated by being a member of the supplied List.

Referenced by antlr3CommonTokenStreamNew().

Function that returns all the tokens indicated by the specified bitset, within a range of tokens.

Referenced by antlr3CommonTokenStreamNew(), getTokenRange(), getTokensList(), and getTokensType().

Function that returns all tokens of a certain type within a range.

Referenced by antlr3CommonTokenStreamNew().

The index into the tokens list of the current token (the next one that will be consumed. p = -1 indicates that the token list is empty.

Referenced by antlr3CommonTokenStreamNew(), consume(), fillBuffer(), getTokens(), getTokensSet(), LB(), reset(), seek(), tindex(), tokLT(), toString(), and toStringSS().

Function that resets the token stream so that it can be reused, but but that does not free up any resources, such as the token factory the factory pool and so on. This prevents the need to keep freeing and reallocating the token pools if the thing you are building is a multi-shot dameon or somethign like that. It is much faster to just reuse all the vectors.

Referenced by antlr3CommonTokenStreamNew().

A simple filter mechanism whereby you can tell this token stream to force all tokens of type ttype to be on channel. For example, when interpreting, we cannot exec actions so we need to tell the stream to force all WS and NEWLINE to be a different, ignored channel.

Referenced by antlr3CommonTokenStreamNew().

Whatever is supplying the COMMON_TOKEN_STREAM needs a pointer to itself so that this can be accessed by any of the API functions which it implements.

Records every single token pulled from the source indexed by the token index. There might be more efficient ways to do this, such as referencing directly in to the token factory pools, but for now this is convenient and the ANTLR3_LIST is not a huge overhead as it only stores pointers anyway, but allows for iterations and so on.

Referenced by antlr3CommonTokenStreamNew(), antlr3CTSFree(), consume(), fillBuffer(), get(), getTokens(), LB(), reset(), size(), skipOffTokenChannels(), skipOffTokenChannelsReverse(), tokLT(), and toStringSS().

The ANTLR3_TOKEN_STREAM interface implementation, which also includes the intstream implementation. We could duplicate the pANTLR_INT_STREAM in this interface and initialize it to a copy, but this could be confusing it just results in one more level of indirection and I think that with judicial use of 'const' later, the optimizer will do decent job.

Referenced by antlr3CommonTokenDebugStreamSourceNew(), antlr3CommonTokenStreamNew(), antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), antlr3RecognitionExceptionNew(), fillBuffer(), getTokensSet(), and skipOffTokenChannels().