Token and TokenStream

Now that you’ve used syn to parse input conveniently, let’s look under the hood at the types that make it all work.

What is a token?

When the Rust compiler reads your source code, the first thing it does is lexing (also called tokenization): breaking the source text into a sequence of tokens. Each token is the smallest meaningful unit of the language.

For example, this Rust code:

#![allow(unused)]
fn main() {
fn add(a: i32) -> i32 { a + 1 }
}

is broken into a tree of tokens:

Ident("fn")
Ident("add")
Group(Parenthesis)
├── Ident("a")
├── Punct(':')
└── Ident("i32")
Punct('-')
Punct('>')
Ident("i32")
Group(Brace)
├── Ident("a")
├── Punct('+')
└── Literal(1)

Notice that this isn’t a flat list — parenthesized and braced portions form nested groups. Each token also has a span (its location in the source file).

TokenTree

The proc_macro crate represents tokens using the TokenTree enum:

#![allow(unused)]
fn main() {
pub enum TokenTree {
    Group(Group),     // delimited group: (...), [...], {...}
    Ident(Ident),     // identifier or keyword: foo, struct, pub
    Punct(Punct),     // punctuation: +, :, #
    Literal(Literal), // literal value: 42, "hello", 3.14
}
}

Notice that delimiters ((, ), [, ], {, }) are not individual tokens. Instead, everything inside a matching pair of delimiters is grouped into a Group. This means the token tree is actually a tree, not a flat list — groups contain other token trees.

TokenStream

A TokenStream is a sequence of TokenTree values. It’s the type that every procedural macro receives and returns.

You can iterate over a TokenStream to inspect its tokens:

#![allow(unused)]
fn main() {
use proc_macro::TokenStream;

#[proc_macro_derive(Inspect)]
pub fn inspect(input: TokenStream) -> TokenStream {
    for tree in input {
        eprintln!("{tree:?}");
    }
    TokenStream::new()
}
}

We use eprintln! rather than println! here. Both format and print text, but println! writes to stdout while eprintln! writes to stderr. Both show up during compilation, so either works for debugging — but by convention proc macros use eprintln!, keeping debug noise on stderr and separate from anything a build might emit on stdout. This makes eprintln! a handy debugging tool alongside cargo expand.

Keep in mind that a proc macro only runs when the crate that invokes it is actually (re)compiled. If a build is cached, the macro doesn’t run and you’ll see no output — touch the source or run cargo clean to force a rebuild.

Descending into groups

Iterating with for tree in input only visits the top-level tokens. Remember that a TokenStream is a tree: anything inside (...), [...], or {...} is wrapped in a Group, and its contents are a nested TokenStream. For a struct like struct Pair { x: i32 }, the top level is just Ident("struct"), Ident("Pair"), and a single Group — the fields x and i32 live inside that group.

To reach them, call Group::stream() to get the inner stream and recurse. Here we extend the Inspect example above to walk the whole tree, printing nested tokens with indentation:

#![allow(unused)]
fn main() {
use proc_macro::{TokenStream, TokenTree};

fn print_tree(stream: TokenStream, depth: usize) {
    let indent = "  ".repeat(depth);
    for tree in stream {
        match tree {
            TokenTree::Group(group) => {
                eprintln!("{indent}Group {:?}", group.delimiter());
                // `group.stream()` is the nested TokenStream — recurse into it.
                print_tree(group.stream(), depth + 1);
            }
            other => eprintln!("{indent}{other:?}"),
        }
    }
}
}

Without the Group arm, print_tree would only ever see the top-level tokens; calling group.stream() is what lets it descend into (...), [...], and {...}.

Building a TokenStream

You can build a TokenStream from individual TokenTree values:

#![allow(unused)]
fn main() {
use proc_macro::{TokenStream, TokenTree, Ident, Span};

let ident = TokenTree::Ident(Ident::new("hello", Span::call_site()));
let stream: TokenStream = [ident].into_iter().collect();
}

Ident represents an identifier or keyword, and Span::call_site() gives it the location of the macro invocation.

Or parse it from a string:

#![allow(unused)]
fn main() {
let stream: TokenStream = "fn foo() {}".parse().unwrap();
}

Building token streams by hand is tedious. That’s why we’ll soon introduce quote!, which makes this much more ergonomic.

Exercise

The exercise for this section is located in 03_basics/04_token_and_tokenstream

Write a derive macro that inspects the token stream it receives and counts the number of Ident tokens in it — including those nested inside groups, which you’ll need to recurse into with Group::stream(). The macro should generate a method that returns this count.