Parser signature to accept tokens

[Flone-dnb]

Hi, I recently started using this project and it's amazing, but I'm currently struggling with parser signatures.

I have a parser that parses string into tokens but now I want to make a new parser that parses tokens into statements. I've looked at examples and discussions and still can't understand what I'm doing wrong because error messages are just unreadable for me.

I have this parser:

pub fn token_parser<'src>(
) -> impl Parser<'src, &'src str, Vec<(Token<'src>, Span)>, extra::Err<Rich<'src, char, Span>>>

and I use it like this:

// Parse tokens.
let (tokens, errors) = parser::token_parser()
    .parse(file_content.as_str())
    .into_output_errors();

// error checking, etc...

let tokens: Vec<(parser::Token<'_>, _)> = tokens.unwrap();

parser::statement_parser().parse(tokens); // this does not want to compile

My statement parser looks like this:

pub fn statement_parser<'src>() -> impl Parser<
    'src,
    Vec<(Token<'src>, Span)>,
    Vec<(Statement<'src>, Span)>,
    extra::Err<Rich<'src, char, Span>>,
> {
    // Sample code:
    any()
        .map_with(|tok, extra| {
            (
                Statement::SomeDeclaration("TODO"),
                extra.span(),
            )
        })
        .repeated()
        .collect()
}

So I get a lot of errors like:

the trait bound `Vec<(parser::Token<'src>, chumsky::span::SimpleSpan)>: ValueInput<'_>` is not satisfied

The tutorial does not seem to cover this case, I've tried using ValueInput from some examples but was still getting lots of errors. Any help on this?

[zesterer]

So, Vec<T> doesn't implement Input, but &[T] does: so you can just use a slice as input and pass the vector by reference and that should work fine, like so:

pub fn statement_parser<'src>() -> impl Parser<
    'src,
    &'src [(Token<'src>, Span)],
    Vec<(Statement<'src>, Span)>,
    extra::Err<Rich<'src, char, Span>>,
> {
    ...
}

One thing to note is that to tell chumsky that your input contains both tokens and spans, you'll need to call .spanned() on the vector, which gives you a more complex type as your input.

At this point, it's easier to just make your parser generic over all input types that have the property you want them to have, like so:

pub fn statement_parser<'src, I>() -> impl Parser<
    'src,
    I,
    Vec<(Statement<'src>, Span)>,
    extra::Err<Rich<'src, char, Span>>,
>
where
    I: Input<'a, Token = Token<'src>, Span = Span>,
{
    ...
}

...

parser::statement_parser().parse(tokens.spanned());

[Flone-dnb]

@zesterer Thank you for a very quick answer! Unfortunately this still does not compile. First, spanned requires an argument, I looked at examples and used this:

parser::statement_parser()
    .parse(tokens.spanned(SimpleSpan::new(0, tokens.last().unwrap().1.end()))); // not sure about this

and the parser still has some errors, I've tried fixing some errors and writing something meaningful like so:

pub fn statement_parser<'src>() -> impl Parser<
    'src,
    &'src [(Token<'src>, Span)],
    Vec<(Statement<'src>, Span)>,
    extra::Err<Rich<'src, (Token<'src>, Span), Span>>,
> {
    let var_type = select! { Token::TypeName(t) => t };
    let var_ident = select! { Token::Ident(ident) => ident };

    let variable_declaration = var_type
        .then(var_ident)
        .then_ignore(just(Token::Op("=")))
        .map(|(t, name)| Statement::VariableDeclaration(t, name));

    variable_declaration
        .map_with(|statement, extra| (statement, extra.span()))
        .repeated()
        .collect()
}

but now I'm getting even more compilation errors, for example:

the trait bound `parser::Token<'_>: OrderedSeq<'_, (parser::Token<'src>, chumsky::span::SimpleSpan)>` is not satisfied

and

type mismatch resolving `<&[(Token<'src>, SimpleSpan)] as Input<'_>>::Token == Token<'_>`
expected tuple `(parser::Token<'src>, chumsky::span::SimpleSpan)`
    found enum `parser::Token<'_>`

and I still can't understand what I'm doing wrong here. Help. 🥲

[zesterer]

First, spanned requires an argument

As mentioned in the docs, spanned expects you to pass it an 'end of input' span. This is a span (usually zero-width, although there's no specific need for that to be the case) that sits just beyond the last element of the input. For example, if you're parsing a string, my_str.len()..my_str.len() would do just fine. The reasons for this are an annoying implementation detail, but it basically comes down to the fact that it's possible for an input to produce no tokens but still need to somehow generate a span.

and the parser still has some errors, I've tried fixing some errors

I mentioned that it's a good idea to make your parser generic over the input type to avoid this sort of thing, as in the example I gave. You can see an example of this sort of thing (i.e: don't copy it word-for-word, exactly what constraints you use depends on what you want to be able to do with your input) here.

The specific error you're seeing is because you've specified the input type as &'src [(Token<'src>, Span)], but this is telling chumsky that the input produces tokens of type (Token<'src>, Span): that's not correct, and it's why you're getting errors. By making your parser generic over the input, you can tell it exactly what token and span to expect instead of having to manually type out the input type.

[Flone-dnb]

@zesterer You are right, making the parser generic fixed the issue! About spanned I was a little confused that I needed to pass something like my_str.len()..my_str.len() but it does the job. Thanks a lot!

[zesterer]

As I say, it's an unfortunate product of the fact that a) chumsky needs to generate spans for you but b) sometimes it has no tokens available to use the spans for, so it needs to have something to fall back on. Great that you've got it working!