ctags-optlib - Man Page

Though it is possible to define a parser from command line, you don't want to type the same command line each time when you need the parser. You can store options for defining a parser into a file.

ctags loads files (preload files) listed in "FILES" section of ctags(1) at program starting up. You can put your parser definition needed usually to the files.

--options=<pathname>, --options-maybe=<pathname>, and --optlib-dir=[+]<directory> are for loading optlib files you need occasionally. See "Option File Options" section of ctags(1) for these options.

As explained in "FILES" section of ctags(1), options for defining a parser listed line by line in an optlib file. Prefixed white spaces are ignored. A line starting with '#' is treated as a comment.  Escaping shell meta character is not needed.

Use .ctags as file extension for optlib file. You can define multiple parsers in an optlib file but it is better to make a file for each parser definition.

--_echo=<msg> and --_force-quit=<num> options are for debugging optlib parser.

1. Design the parser

   You need know both the target language and the ctags' concepts (definition, reference, kind, role, field, extra). About the concepts, ctags(1) of Universal Ctags may help you.

2. Give a name to the parser

   Use --langdef=<name> option. <name> is referred as <LANG> in the later steps.

3. Give a file pattern or file extension for activating the parser

   Use --map-<LANG>=[+|-]<extension>|<pattern>.

4. Define kinds

   Use --kinddef-<LANG>=<letter>,<name>,<description> option. Universal Ctags introduces this option.  Exuberant Ctags doesn't have. In Exuberant Ctags, a kind is defined as a side effect of specifying --regex-<LANG>= option. So user doesn't have a chance to recognize how important the definition of kind.

5. Define patterns

   Use --regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/[<flags>] option for a single-line regular expression. You can also use --mline-regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/{mgroup=<N>}[<flags>] option for a multi-line regular expression.

   As <kind-spec>, you can use the one-letter flag defined with --kinddef-<LANG>=<letter>,<name>,<description> option.

You can specify more than one flag, <letter>|{<name>}, at the end of --regex-<LANG> to control how Universal Ctags uses the pattern.

Exuberant Ctags uses a <letter> to represent a flag. In Universal Ctags, a <name> surrounded by braces (name form) can be used in addition to <letter>. The name form makes a user reading an optlib file easier.

The most of all flags newly added in Universal Ctags don't have the one-letter representation. All of them have only the name representation. --list-regex-flags lists all the flags.

basic (one-letter form b)

The pattern is interpreted as a POSIX basic regular expression.

exclusive (one-letter form x)

Skip testing the other patterns if a line is matched to this pattern. This is useful to avoid using CPU to parse line comments.

extend (one-letter form e)

The pattern is interpreted as a POSIX extended regular expression (default).

pcre2 (one-letter form p, experimental)

The pattern is interpreted as a PCRE2 regular expression explained in pcre2syntax(3).  This flag is available only if the ctags is built with pcre2 library. See the output of --list-features option to know whether your ctags is built-with pcre2 or not.

icase (one-letter form i)

The regular expression is to be applied in a case-insensitive manner.

placeholder

Don't emit a tag captured with a regex pattern.  The replacement can be an empty string.  See the following description of scope=... flag about how this is useful.

postrun

Match the pattern at the end of all the parsing processes, including:

• running the built-in code for <LANG>,
• applying --mline-regex-<LANG> patterns,
• applying --_mtable-regex-<LANG> patterns, and
• applying non-postrun --regex-<LANG>= patterns.

This flag is helpful when combined with scope=intervaltab.

The built-in code processes source files line-by-line delivered by the main part of ctags.  The main part applies non-postrun --regex-<LANG>= patterns to a line just after delivering the line to the code of built-in code. Thus, non-postrun --regex-<LANG>= patterns cannot refer to the tags information finally extracted by the built-in code.

This is where the postrun comes into play. The main part never applies postrun --regex-<LANG>= patterns when delivering lines to the code of built-in code. Instead, it applies the postrun patterns in batch after delivering all lines to the built-in code. The postrun patterns can refer to the tags information extracted by the built-in code.

--mline-regex-<LANG> and --_mtable-regex-<LANG> have no {postrun} flag because the main part always applies the patterns specified with the options after running the built-in code for <LANG>.

See also the description of scope=intervaltab flag.

scope=intervaltab

Use the interval table maintaind by the main part of ctags to fill in the scope: field. This flag is useful for extending a built-in parser with the --regex-<LANG>= option with postrun flag.

The interval table holds tag entries having both line: and end: fields. These tag entries are stored in a table keyed by their line: and end: field pairs. Therefore, the table can answer queries like, "Is there a tag entry that includes this line?" or "Which tag entry contains this line?"

The source line, where postrun --regex-<LANG> pattern finds a language object, can be a key for such queries. The tag entry returned by the table is set in the scope: field of the newly created tag entry for the language object.

postrun flag is needed for running the built-in parser that stores tag entries before applying patterns specified with --regex-<LANG>.

See also the example in "Using the interval table".

scope=(ref|push|pop|clear|set|replace)

Specify what to do with the internal scope stack.

A parser programmed with --regex-<LANG> has a stack (scope stack) internally. You can use it for tracking scope information. The scope=... flag is for manipulating and utilizing the scope stack.

If {scope=push} is specified, a tag captured with --regex-<LANG> is pushed to the stack. {scope=push} implies {scope=ref}.

You can fill the scope field (scope:) of captured tag with {scope=ref}. If {scope=ref} flag is given, ctags attaches the tag at the top to the tag captured with --regex-<LANG> as the value for the scope: field.

ctags pops the tag at the top of the stack when --regex-<LANG> with {scope=pop} is matched to the input line.

Specifying {scope=clear} removes all the tags in the scope. Specifying {scope=set} removes all the tags in the scope, and then pushes the captured tag as {scope=push} does.

{scope=replace} does the three things sequentially. First it does the same as {scope=pop}, then fills the scope: field of the tag captured with --regex-<LANG>, and pushes the tag to the scope stack as if {scope=push} was given finally. You cannot specify another scope action together with {scope=replace}.

You don't want to specify {scope=pop}{scope=push} as an alternative to {scope=replace}; {scope=pop}{scope=push} fills the scope: field of the tag captured with --regex-<LANG> first, then pops the tag at the top of the stack, and pushes the captured tag to the scope stack finally. The timing when filling the end field is different between {scope=replace} and {scope=pop}{scope=push}.

In some cases, you may want to use --regex-<LANG> only for its side effects: using it only to manipulate the stack but not for capturing a tag. In such a case, make <name_pattern> component of --regex-<LANG> option empty while specifying {placeholder} as a regex flag. For example, a non-named tag can be put on the stack by giving a regex flag "{scope=push}{placeholder}".

You may wonder what happens if a regex pattern with {scope=ref} flag matches an input line but the stack is empty, or a non-named tag is at the top. If the regex pattern contains a {scope=ref} flag and the stack is empty, the {scope=ref} flag is ignored and nothing is attached to the scope: field.

If the top of the stack contains an unnamed tag, ctags searches deeper into the stack to find the top-most named tag. If it reaches the bottom of the stack without finding a named tag, the {scope=ref} flag is ignored and nothing is attached to the scope: field.

When a named tag on the stack is popped or cleared as the side effect of a pattern matching, ctags attaches the line number of the match to the end: field of the named tag.

ctags clears all of the tags on the stack when it reaches the end of the input source file. The line number of the end is attached to the end: field of the cleared tags.

warning=<message>

print the given <message> at WARNING level

fatal=<message>

print the given <message> and exit

mgroup=<N>

decide the location of the tag extracted with --mline-regex-<LANG> option.

<N> is the number of a capture group in the pattern, which is used to record the line number location of the tag. mgroup=<N> flag is not an optional. You must add an mgroup=<N> flag, even if the <N> is 0 (meaning the start position of the whole regex pattern).

scope=intervaltab

See the description for the flag in "FLAGS FOR --regex-<LANG> OPTION".

Unlike --regex-<LANG> option, you don't have to specify postrun flag.

This is the definition (pod.ctags) used in ctags for parsing Pod (https://perldoc.perl.org/perlpod.html) file.

--langdef=pod
--map-pod=+.pod

--kinddef-pod=c,chapter,chapters
--kinddef-pod=s,section,sections
--kinddef-pod=S,subsection,subsections
--kinddef-pod=t,subsubsection,subsubsections

--regex-pod=/^=head1[ \t]+(.+)/\1/c/
--regex-pod=/^=head2[ \t]+(.+)/\1/s/
--regex-pod=/^=head3[ \t]+(.+)/\1/S/
--regex-pod=/^=head4[ \t]+(.+)/\1/t/

Let's think about writing a parser for a very small subset of the Ruby language.

input source file (input.srb):

class Example
  def methodA
        puts "in class_method"
  end
  def methodB
        puts "in class_method"
  end
end

The parser for the input should capture Example with class kind, methodA, and methodB with method kind. methodA and methodB should have Example as their scope. end: fields of each tag should have proper values.

optlib file (sub-ruby.ctags):

--langdef=subRuby
--map-subRuby=.srb
--kinddef-subRuby=c,class,classes
--kinddef-subRuby=m,method,methods
--regex-subRuby=/^class[ \t]+([a-zA-Z][a-zA-Z0-9]+)/\1/c/{scope=push}
--regex-subRuby=/^end///{scope=pop}{placeholder}
--regex-subRuby=/^[ \t]+def[ \t]+([a-zA-Z][a-zA-Z0-9_]+)/\1/m/{scope=push}
--regex-subRuby=/^[ \t]+end///{scope=pop}{placeholder}

command line and output:

$ ctags --quiet --fields=+eK \
--options=./sub-ruby.ctags -o - input.srb
Example input.srb       /^class Example$/;"     class   end:8
methodA input.srb       /^  def methodA$/;"     method  class:Example   end:4
methodB input.srb       /^  def methodB$/;"     method  class:Example   end:7

Let's try to extract kernel threads defined in Linux kernel.

kthread_run is the function for creating and starting a kernel thread. For example, kswapd kernel thread is created with the function like (quoted from linux/mm/vmscan.c):

void kswapd_run(int nid)
{
                /* ... */
                pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
                /* ... */

kthread-v1.ctags illustrates the way to extract the name of kernel threads appeared at the third argument of kthread_run.

kthread-v1.ctags:

--kinddef-C=K,kernelThread,the name of kernel thread in Linux kernel
--regex-C=/kthread_run\([^"]+"([^"]+)"/\1/K/

With kthread-v1.ctags, ctags emits the following tags:

$ ctags --options=linux/kthread-v1.ctags  -o - vmscan.c
...
kswapd%d        vmscan.c        /^... = kthread_run(kswapd, pgdat, "kswapd%d"...$/;"    K
...

Using the interval table, you can attach the name of the function where the kernel thread is defined.

kthread-v2.ctags:

--kinddef-C=K,kernelThread,the name of kernel thread in Linux kernel
--regex-C=/kthread_run\([^"]+"([^"]+)"/\1/K/{postrun}{scope=intervaltab}

With kthread-v2.ctags, ctags emits the following tags:

$ ctags --options=linux/kthread-v2.ctags  -o - vmscan.c
...
kswapd%d        vmscan.c        /^... = kthread_run(kswapd, pgdat, "kswapd%d"...$/;"    K       function:kswapd_run
...

With the new .ctags file, function:kswapd_run is attach to the tag entry as its scope field.