Implement Top level Language Parser

[chase-metzger]

Right now I'm trying to implement a simple language parser for a toy language I made up.

Looks something like:

1 + 2; func test(a, b) { return a + b; }
I want to generate a list (aka a vector) of Expr's which is defined very similar to how json_value is defined in the json example.
So the first element in the parsed list would be the 1 + 2 and the second would be the function defintion.

Right now I have a dsl::list(dsl::p<top-level-production-type>, dsl::sep(dsl::newline / dsl::semicolon) as what the rule is for the final production in the grammar.
And the value field is lexy::as_list<ast::ExprList>

The result I am currently getting is only getting the first element parsed. So just 1 +2 or func test(...) if the that's what's first.
How can I modify it so that it collects each type of production/ast type into a list of them?

[foonathan]

That should be the behavior that is already happening. Can you post a minimal example? Preferable on the playground: https://lexy.foonathan.net/playground/

[chase-metzger]

https://lexy.foonathan.net/playground/?id=xe1eKx3bs&mode=tree

Here's the most minimal and stripped down example that illustrates the problem.
The final production only captures the func_def and not the number after it. But if you add a semicolon to the end of the function (ie after the closing bracket) it will capture the number and there will be another node in the tree. I'm suspecting it has something to do with whitespace. How I would like it to act is after each successful parse of a wasd_value to have that be a node in the list. Sometimes, like after a function, there shouldn't need to be a semicolon. But after a number or boolean there should at least be a newline character (optionally a semicolon).

[foonathan]

Yes, the issue has to do with whitespace handling. If newlines are significant to your grammar in any way, they can't be part of automatic whitespace skipping. The fix is to add a single top-level whitespace directive that is dsl::ascii::blank (i.e. not newline), then you can use dsl::while_one(dsl::ascii::newline) (one or more newlines) as separator. I'm assuming newlines are then allowed e.g. inside a function, right? In that case, instead of using dsl::recurse<wasd_value> you add a helper transparent production that overrides the whitespace to dsl::ascii::space as newlines are now allowed: https://lexy.foonathan.net/playground/?id=ja8hK6s8q&mode=tree (I've changed production, block, and removed all other whitespace members)

BTW: dsl::ascii::space / dsl::ascii::newline is unecessary, space includes newline. blank is the one that doesn't.

[chase-metzger]

This seems to almost work. When the function body is empty it fully parses the input, but gives a parse error at the ending curly brace of any empty function

[foonathan]

Ah, the issue is that an empty list doesn't skip newlines. We can fix it, by parsing a non-optional list, but where the item can be empty if it reached the closing }: https://lexy.foonathan.net/playground/?id=8oT1549E9&mode=tree

This isn't entirely ideal though, I might add some support for .opt_list(item, sep, inner_whitespace_rule) or something like that.

[foonathan]

Nevermind my previous answer. I found a solution that is a lot simpler: https://lexy.foonathan.net/playground/?id=rsaecr3nr&mode=tree

Don't allow newlines at the top-level and use them as separator, do allow newlines inside a func_def by overriding whitespace there. No other changes necessary.

[chase-metzger]

So I finally got around to playing with these and experimenting with them to try and fix the issues with each one.

The first answer (before this one) does pretty much what I would like in the playground, but I can't compile locally it with lexy::parse.
I get a error about cannot instantiate ast::Expr (my top-level variant AST type) with a lexy::nullopt. lexy::parse_as_tree works fine because it doesn't do value production. I don't see a way to fix this.

The second answer (which does compile locally with lexy::parse) has a weird thing where if there's a number first and a function second
in the input, then it adds both to the parse tree but not any more after. But if the function is first, then the number is not added unless there's a semicolon and the number is right after it with not newline.
I tried to fix this by playing the the whitespace but only ended back to where it parses everything success fully (parse and parse tree versions) but gives a parse error for empty grammar::blocks.

[foonathan]

I get a error about cannot instantiate ast::Expr (my top-level variant AST type) with a lexy::nullopt. lexy::parse_as_tree works fine because it doesn't do value production. I don't see a way to fix this.

You can add a constructor to ast::Expr that accepts lexy::nullopt and produces a null expression (I'm assuming func f() {} is the same as func f() { null }?). Alternatively, you can write your own callback.

The second answer (which does compile locally with lexy::parse) has a weird thing where if there's a number first and a function second

Urgh, yes, it skips any newlines after the closing } of a function. I don't have any good idea of fixing that one.

(BTW: by adding + dsl::eof to the top-level production, you get a parse error instead of having it silently ignore input.)

Given that it is a toy language, you made up: can't you just make semicolons mandatory? IMO it is not a good idea to have newlines sometimes be significant tokens and sometimes insignificant whitespace.

[foonathan]

Okay, this is the one I think: https://lexy.foonathan.net/playground/?id=8xvo6zhvM&mode=tree

The idea is again to override whitespace while parsing a nested value. However, this doesn't handle func f() {\n} or func f(){\nnull;}, as whitespace is only skipped after something. So instead of dsl::curly_bracketed we can define or own brackets that skip whitespace after the initial {. It's not elegant, but it works.

You need to do something similar if you want newlines in the parameters as well.

[chase-metzger]

This works perfect! It even works for parsing blocks at the top-level and for blocks connected to other statements like if.
Elegance is not really too much of an issue. I think it's much better than what I was experimenting with the last couple days.

I added a whitespace field to my func_proto production and that seems to fix any issue with spaces or newlines, which I really only care about the parameter list for spaces and newlines.

Thanks for your help. I want to say I'm glad I found this library. Even though (given that it's C++) compile errors can be abstract with all the template meta-programming involved in it, I still think its a unique and quite elegant approach to building lexers.

[foonathan]

Thanks! I haven't given error messages too much care so far. If there are any that are particularly unreadable, please file an issue and I'll see what I can do to polish it before the initial release.