Preliminary work to parse a "pythonic" language and translate it to C

[jokoon]

My goal is a language that gets translated to C, to benefit from existing C compilers and libraries.

That would be some sort of better C, with python indentation, strings, lists, dict and tuple-as-struct. I'm new to parsing so this is a training project.

I used lexy to write a first draft of a parser. It might have flaws, but I'm thankful to foonathan for the help :)

The first step was to tokenize python style indents, I did it using a python script:

    def leading_spaces(s):
        spaces = 0
        for c in s:
            if c == ' ':
                spaces += 1
            else:
                break
        return spaces//4

    def token_indenter(source, endline, indents = ('{', '}'), print_diagnose = False):

        indent_token, dedent_token = indents
        # adding \n to improve the thing
        if source[-2:] != '\n\n':
            print('added\\n')
            source += '\n\n'
        lines = [line for line in source.split('\n')]

        # counting indenting space (no tab)
        lines = [(line, leading_spaces(line))
            for line in lines]

        lines = [(
            line,
            (lines[i-1][1] if line == '' else ldspc)
                if 1<=i<len(lines) else 0)
            for (i, (line, ldspc)) in enumerate(lines)]

        # prev None . . . .
        # next  . . . . None

        # assigning prev/next line count for each line
        lines = [
            (
                lines[i-1][1] if 1<=i<len(lines) else None, # prev: 1 to len-1
                lines[i+1][1] if 0<=i<len(lines)-1 else None, # next: 0 to len-2
                line, spaces
             )
            for (i, (line, spaces)) in enumerate(lines)]

        # difference of spaces between lines
        lines = [
            (spaces - prev if prev!=None else None,
            spaces - nextt if nextt!=None else None,
            line, spaces)
            for (prev, nextt, line, spaces) in lines]

        lines_tokens = []
        # we only insert tokens on the same line to keep the context, hence redundancy

        # generating indent/dedent tokens
        for (diffprev, diffnext, line, spaces) in lines:
            lines_tokens.append((
                line,
                diffprev, diffnext,
                1 if diffprev == 1 else 0, # indent
                diffnext if diffnext else 0, # dedent
                spaces,
                diffnext >= 0
                # True
                    if diffnext != None and line
                    else False, # endline
                ))

        laidout = ''
        laidout_better = ''
        dent_width = max(len(indent_token), len(dedent_token)) + 1
        for (line, diffprev, diffnext, indent, dedent, spaces, newline) in lines_tokens:

            # indent_tok = '{' if indent else ''
            # dedent_tok = '}'*dedent if dedent else ''

            indent_tok = indent_token if indent else ''
            dedent_tok = (dedent_token+' ') * dedent if dedent else ''
            spaces_brace = spaces - 1
            spaces *= 4
            spaces_brace *= 4
            cool_line = (
                (((' '*spaces_brace) + indent_tok + '\n') if indent_tok else '')
                +f'{line if line else " "*spaces} {endline if newline else ""}\n'
                # +f'{line if line else " "*spaces}{endline}\n'
                +(((' '*spaces_brace) + dedent_tok + '\n') if dedent_tok else '')
                )

            laidout += f'{indent_tok} {line} {dedent_tok}'

            diagnose = (f'dfpv {str(diffprev):>5} dfnx {str(diffnext):>5} {indent_tok:12} {dedent_tok:12} | {repr(line)}')
            laidout_better += cool_line

            if print_diagnose:
                print(diagnose)
        return laidout_better

Of course this code sample makes no sense, but I'm planning to rely on C errors.

The parser is about 300 lines of lexy rules:

    #include <cstdio>
    #include <lexy/action/parse.hpp>
    #include <lexy/action/parse_as_tree.hpp>
    #include <lexy/callback.hpp>
    #include <lexy/dsl.hpp>
    #include <lexy_ext/compiler_explorer.hpp>
    #include <lexy_ext/report_error.hpp>
    #include <vector>
    #include <string>
    #include <iostream>
    #include <sstream>
    #include <iterator>
    #include <fstream>
    #include <map>
    // using namespace std;

    /*
    lexy::match -> true or false
    lexy::validate -> explains why it did not match
    Automatic whitespace skipping is done by adding a static constexpr auto whitespace member to the root production.
    while_ must be given a branch rule
    */

    // dump
        // Functions can only have a single argument for simplicity.
        // auto var_or_call = dsl::p<name> >> dsl::if_(sparen_expr);
        // auto literal     = dsl::p<integer>;
        // auto paren_expr = dsl::parenthesized(dsl::p<nested_expr>);


    namespace {

    namespace grammar_clak {
        namespace dsl = lexy::dsl;

        struct identifier {
            static constexpr auto rule = []{
                auto head = dsl::ascii::alpha_underscore;
                auto tail = dsl::ascii::alpha_digit_underscore;
                return dsl::identifier(head, tail);
            }();
        };

        struct expression2;
        // nested_expr is only used by expression2 as an atom, for recursivity
        struct nested_expr : lexy::transparent_production {
            // We change the whitespace rule to allow newlines:
            // as it's nested, the REPL can properly handle continuation lines.
            static constexpr auto whitespace = dsl::ascii::space; // | escaped_newline;

            // The rule itself just recurses back to expression, but with the adjusted whitespace now.
            static constexpr auto rule = dsl::recurse<struct expression2>;
        };

        struct paren_expr{
            // corresponds to "paren_or_tuple" that was suggested
            static constexpr auto rule =
                dsl::parenthesized.list(dsl::p<nested_expr>, dsl::sep(dsl::comma));
        };


        struct expression2 : lexy::expression_production {
            // order: || && | ^ & (== !=) (< > <= >=) (<< >>) (+ -) (* / %)

            static constexpr auto whitespace = dsl::ascii::space;
            // static constexpr auto atom       = dsl::integer<int>;

            struct expected_operand { static constexpr auto name = "expected operand"; };
            // We need to specify the atomic part of an expression.
            static constexpr auto atom = [] {
                // shouldn't use dsl::p<expression2> instead dsl::p<nested_expr>
                auto var_or_call = dsl::p<identifier> >> dsl::if_(dsl::p<paren_expr>);
                return
                    dsl::p<paren_expr>
                    | var_or_call
                    | dsl::integer<int>
                    | dsl::error<expected_operand>;
            }();

            struct prefix : dsl::prefix_op {
                static constexpr auto op = dsl::op(dsl::lit_c<'-'>);
                // static constexpr auto op = dsl::op(dsl::lit_c<'-'>) / dsl::op(dsl::lit_c<'~'>);
                using operand            = dsl::atom;
            };
            struct product : dsl::infix_op_left {
                static constexpr auto op =
                    dsl::op(dsl::lit_c<'*'>)
                    / dsl::op(dsl::lit_c<'/'>)
                    / dsl::op(dsl::lit_c<'%'>);

                using operand            = prefix;
            };
            struct sum : dsl::infix_op_left {
                static constexpr auto op =
                dsl::op(dsl::lit_c<'+'>)
                / dsl::op(dsl::lit_c<'-'>);

                using operand            = product;
            };

            struct bit_shift : dsl::infix_op_left {
                static constexpr auto op =
                    dsl::op(LEXY_LIT(">>"))
                    / dsl::op(LEXY_LIT("<<"));

                using operand            = sum;
            };
            struct inequality : dsl::infix_op_left {
                static constexpr auto op =
                    dsl::op(LEXY_LIT(">"))
                    / dsl::op(LEXY_LIT("<"))
                    / dsl::op(LEXY_LIT("<="))
                    / dsl::op(LEXY_LIT(">="));

                using operand            = bit_shift;
            };
            struct equality : dsl::infix_op_left {
                static constexpr auto op =
                dsl::op(LEXY_LIT("=="))
                / dsl::op(LEXY_LIT("!="));

                using operand            = inequality;
            };
            struct bit_and  : dsl::infix_op_left { static constexpr auto op =
                dsl::op(dsl::lit_c<'&'>); using operand = equality;};
            struct bit_xor  : dsl::infix_op_left { static constexpr auto op =
                dsl::op(dsl::lit_c<'^'>); using operand = bit_and;};
            struct bit_or   : dsl::infix_op_left { static constexpr auto op =
                dsl::op(dsl::lit_c<'|'>); using operand = bit_xor;};
            struct bool_and : dsl::infix_op_left { static constexpr auto op =
                dsl::op(LEXY_LIT("&&")); using operand = bit_or;};
            struct bool_or  : dsl::infix_op_left { static constexpr auto op =
                dsl::op(LEXY_LIT("||")); using operand = bool_and;};

            using operation = bool_or; // the expression starts here
        };
        struct paren_or_tuple{ static constexpr auto rule =
            dsl::parenthesized.list(dsl::p<expression2>, dsl::sep(dsl::comma));
        };

        // statements
        struct statement;
        struct expression_dummy {
            static constexpr auto rule = LEXY_LIT("__EXPRESSION__");
        };
        struct expression_stmt {
            static constexpr auto rule =
                dsl::p<expression_dummy> >> LEXY_LIT("__ENDLINE__");
        };
        struct compound_stmt {
            static constexpr auto rule =
                lexy::dsl::brackets(LEXY_LIT("__INDENT__"),
                    LEXY_LIT("__DEDENT__")).list(dsl::recurse<statement>);
        };
        struct else_stmt {
            static constexpr auto rule =
                LEXY_LIT("else") >> LEXY_LIT(":") + dsl::p<compound_stmt>;
        };
        struct elif_stmt {
            // unused!
            static constexpr auto rule =
                LEXY_LIT("elif") >> LEXY_LIT(":") + dsl::p<compound_stmt>;
        };
        struct while_stmt {
            static constexpr auto rule =
                LEXY_LIT("while") >> dsl::p<expression2>
                + LEXY_LIT(":") + dsl::p<compound_stmt>;
        };
        struct for_stmt {
            static constexpr auto rule =
                LEXY_LIT("for") >> dsl::p<expression2>
                + LEXY_LIT(";") + dsl::p<expression2>
                + LEXY_LIT(";") + dsl::p<expression2>
                + LEXY_LIT(":") + dsl::p<compound_stmt>;
        };
        struct if_stmt {
            static constexpr auto rule =
                LEXY_LIT("if") >> dsl::p<expression2>
                + LEXY_LIT(":") + dsl::p<compound_stmt>
                // please implement this
                // + dsl::opt(dsl::p<elif_stmt> | dsl::p<else_stmt>)
                + dsl::opt(dsl::p<else_stmt>);
        };

        struct continue_stmt {
            static constexpr auto rule =
            LEXY_LIT("continue") >> LEXY_LIT("__ENDLINE__");
        };

        struct break_stmt {
            static constexpr auto rule =
            LEXY_LIT("break")    >> LEXY_LIT("__ENDLINE__"); };

        struct return_stmt {
            static constexpr auto rule =
            LEXY_LIT("return")   >> LEXY_LIT("__ENDLINE__"); };

        struct return_stmt_value {
            static constexpr auto rule =
                LEXY_LIT("return")
                >> dsl::p<expression2>
                + LEXY_LIT("__ENDLINE__");
        };

        struct jump_stmt {
            static constexpr auto rule =
                dsl::p<continue_stmt>
                | dsl::p<break_stmt>
                | dsl::p<return_stmt>
                | dsl::p<return_stmt_value>;
        };

        struct assignment_operator {
            static constexpr auto rule = dsl::literal_set(
                LEXY_LIT("="),
                LEXY_LIT("*="), LEXY_LIT("/="), LEXY_LIT("%="),
                LEXY_LIT("+="), LEXY_LIT("-="),
                LEXY_LIT(">>="), LEXY_LIT("<<="),
                LEXY_LIT("&="), LEXY_LIT("|="), LEXY_LIT("^=")
                );
        };
        struct assignment_stmt {
            static constexpr auto rule =
                dsl::p<identifier>
                >> dsl::p<assignment_operator>
                + dsl::p<expression2>
                // + dsl::p<paren_or_tuple>
                + LEXY_LIT("__ENDLINE__");
        };

        struct statement {
            static constexpr auto whitespace = dsl::ascii::space;
            static constexpr auto rule =
                dsl::p<compound_stmt>
                | dsl::p<if_stmt>
                | dsl::p<expression_stmt>
                | dsl::p<while_stmt>
                | dsl::p<for_stmt>
                | dsl::p<jump_stmt>
                | dsl::p<assignment_stmt>
                ;
        };

        struct string_literal {
            static constexpr auto rule = [] {
                // Arbitrary code points that aren't control characters.
                auto c = -dsl::ascii::control;
                return dsl::quoted(c);
            }();
        };

        struct float_literal {
            // not just a float, also an int (lexy can't differentiate)
            struct period_opt_digits { static constexpr auto rule =
                dsl::period >> dsl::opt(dsl::digits<>); };

            static constexpr auto rule =
                dsl::opt(dsl::lit_c<'-'>)
                +(dsl::digits<> >> dsl::opt(dsl::p<period_opt_digits>)
                | dsl::period >> dsl::digits<>)
                ;
        };
        struct number : lexy::token_production {
            // A signed integer parsed as int64_t.
            struct integer : lexy::transparent_production {
                static constexpr auto rule
                    = dsl::minus_sign + dsl::integer<std::int64_t>(dsl::digits<>.no_leading_zero());
            };

            // The fractional part of a number parsed as the string.
            struct fraction : lexy::transparent_production {
                static constexpr auto rule  = dsl::lit_c<'.'> >> dsl::capture(dsl::digits<>);
            };

            // The exponent of a number parsed as int64_t.
            struct exponent : lexy::transparent_production {
                static constexpr auto rule = [] {
                    auto exp_char = dsl::lit_c<'e'> | dsl::lit_c<'E'>;
                    return exp_char >> dsl::sign + dsl::integer<std::int16_t>;
                }();
            };

            static constexpr auto rule
                = dsl::peek(dsl::lit_c<'-'> / dsl::digit<>)
                  >> dsl::p<integer> + dsl::opt(dsl::p<fraction>) + dsl::opt(dsl::p<exponent>);
        };
    }


    } // namespace

    #include <regex>
    std::string unindent(std::string s, int n){

        std::string indents(n*4, ' ');
        return std::regex_replace(
            s,
            std::regex(indents), "");
    }

    int main(int n, char * argv[])
    {
        using namespace std;

        if(n == 2) {
            std::string indentized;

            std::ifstream ifs(argv[1]);
            if(ifs.is_open()){
                indentized = std::string(
                    (std::istreambuf_iterator<char>(ifs)),
                    (std::istreambuf_iterator<char>()));
            }
            else{
                std::cout << "could not open " << argv[1] <<  std::endl;
                return -1;
            }

            // lexy objects
            lexy::buffer<lexy::utf8_encoding> input(indentized);
            lexy::parse_tree_for<decltype(input)> tree;

            lexy::parse_as_tree<grammar_clak::statement>(tree, input, lexy_ext::report_error);

            // 3 different output formats
            std::ofstream of("parse_tree.nogit.json");
            std::ofstream of_raw("raw_tree.nogit.txt");
            std::ofstream of_tree_fancy("tree_fancy.nogit.txt");

            std::string str_for_fancy;
            // lexy::visualize(std::back_inserter(str_for_fancy), tree, {lexy::visualize_fancy});
            lexy::visualize_to(std::back_inserter(str_for_fancy), tree,
                // {lexy::visualize_use_symbols | lexy::visualize_space});
                // {lexy::visualize_use_symbols | lexy::visualize_use_unicode,});
                {
                    lexy::visualize_use_symbols
                    | lexy::visualize_use_unicode
                    | lexy::visualize_space
                });

            of_tree_fancy << str_for_fancy;


            // hacky way of generate something that looks like json
            int indent = 0;
            auto spaces = [](int indent) {return std::string(indent*4,' ');};

            for (auto [event, node] : tree.traverse())
            {
                switch (event)
                {
                    case lexy::traverse_event::enter:
                        of << spaces(indent) << "{ \"" << node.kind().name() << "\": [" << std::endl;
                        of_raw << "enter:" << node.kind().name() << std::endl;
                        indent+=1;
                        break;

                    case lexy::traverse_event::exit:
                        of_raw << "exit:" << node.kind().name() << std::endl;

                        indent-=1;
                        of << spaces(indent) << "], }," << std::endl;
                        break;
                    case lexy::traverse_event::leaf: {
                            std::string s;
                            lexy::visualize_to(std::back_inserter(s), node.lexeme());

                            of_raw << "leaf:" << s << std::endl;
                            of << spaces(indent) << ("\"" + s + "\",") << std::endl;
                            break;
                        }
                }
            }
        }
        else {
            std::cout << "give me a filename" << std::endl;
        }

        return  0;
    }

Here is the pretty tree output. It's possible to remove whitespace noise with some CTRL F:

    statement:
    ├──whitespace: ␣⏎
    └──while_stmt:
       ├──literal: while
       ├──whitespace: ␣
       ├──expression2:
       │  └──identifier:
       │     └──identifier: var
       ├──literal: :
       ├──whitespace: ␣⏎
       └──compound_stmt:
          ├──literal: __INDENT__
          ├──whitespace: ⏎␣␣␣␣
          ├──statement:
          │  └──jump_stmt:
          │     └──continue_stmt:
          │        ├──literal: continue
          │        ├──whitespace: ␣
          │        ├──literal: __ENDLINE__
          │        └──whitespace: ⏎␣␣␣␣
          ├──statement:
          │  └──expression_stmt:
          │     ├──expression_dummy:
          │     │  ├──literal: __EXPRESSION__
          │     │  └──whitespace: ␣
          │     ├──literal: __ENDLINE__
          │     └──whitespace: ⏎␣␣␣␣
          ├──statement:
          │  └──if_stmt:
          │     ├──literal: if
          │     ├──whitespace: ␣
          │     ├──expression2:
          │     │  └──digits: 1
          │     ├──literal: :
          │     ├──whitespace: ␣⏎␣␣␣␣
          │     └──compound_stmt:
          │        ├──literal: __INDENT__
          │        ├──whitespace: ⏎␣␣␣␣␣␣␣␣
          │        ├──statement:
          │        │  └──assignment_stmt:
          │        │     ├──identifier:
          │        │     │  ├──identifier: duh
          │        │     │  └──whitespace: ␣
          │        │     ├──assignment_operator:
          │        │     │  ├──literal: =
          │        │     │  └──whitespace: ␣
          │        │     ├──expression2:
          │        │     │  └──paren_expr:
          │        │     │     ├──literal: (
          │        │     │     ├──expression2:
          │        │     │     │  └──digits: 43
          │        │     │     ├──literal: ,
          │        │     │     ├──whitespace: ␣
          │        │     │     ├──expression2:
          │        │     │     │  ├──identifier:
          │        │     │     │  │  └──identifier: call
          │        │     │     │  └──paren_expr:
          │        │     │     │     ├──literal: (
          │        │     │     │     ├──expression2:
          │        │     │     │     │  └──digits: 2
          │        │     │     │     ├──literal: ,
          │        │     │     │     ├──whitespace: ␣
          │        │     │     │     ├──expression2:
          │        │     │     │     │  └──digits: 5
          │        │     │     │     └──literal: )
          │        │     │     ├──literal: )
          │        │     │     └──whitespace: ␣
          │        │     ├──literal: __ENDLINE__
          │        │     └──whitespace: ⏎␣␣␣␣␣␣␣␣
          │        ├──statement:
          │        │  └──assignment_stmt:
          │        │     ├──identifier:
          │        │     │  ├──identifier: ret
          │        │     │  └──whitespace: ␣
          │        │     ├──assignment_operator:
          │        │     │  ├──literal: =
          │        │     │  └──whitespace: ␣
          │        │     ├──expression2:
          │        │     │  ├──identifier:
          │        │     │  │  └──identifier: fun
          │        │     │  └──paren_expr:
          │        │     │     ├──literal: (
          │        │     │     ├──expression2:
          │        │     │     │  └──digits: 9
          │        │     │     ├──literal: )
          │        │     │     └──whitespace: ␣
          │        │     ├──literal: __ENDLINE__
          │        │     └──whitespace: ⏎␣␣␣␣␣␣␣␣
          │        ├──statement:
          │        │  └──assignment_stmt:
          │        │     ├──identifier:
          │        │     │  ├──identifier: thing
          │        │     │  └──whitespace: ␣
          │        │     ├──assignment_operator:
          │        │     │  ├──literal: =
          │        │     │  └──whitespace: ␣
          │        │     ├──expression2:
          │        │     │  └──identifier:
          │        │     │     ├──identifier: stuff
          │        │     │     └──whitespace: ␣
          │        │     ├──literal: __ENDLINE__
          │        │     └──whitespace: ⏎␣␣␣␣␣␣␣␣␣⏎␣␣␣␣
          │        ├──literal: __DEDENT__
          │        └──whitespace: ␣⏎␣␣␣␣
          ├──statement:
          │  └──assignment_stmt:
          │     ├──identifier:
          │     │  ├──identifier: thing
          │     │  └──whitespace: ␣
          │     ├──assignment_operator:
          │     │  ├──literal: =
          │     │  └──whitespace: ␣
          │     ├──expression2:
          │     │  └──paren_expr:
          │     │     ├──literal: (
          │     │     ├──expression2:
          │     │     │  └──digits: 1
          │     │     ├──literal: ,
          │     │     ├──whitespace: ␣
          │     │     ├──expression2:
          │     │     │  └──digits: 34
          │     │     ├──literal: )
          │     │     └──whitespace: ␣
          │     ├──literal: __ENDLINE__
          │     └──whitespace: ⏎␣␣␣␣
          ├──statement:
          │  └──assignment_stmt:
          │     ├──identifier:
          │     │  ├──identifier: something
          │     │  └──whitespace: ␣
          │     ├──assignment_operator:
          │     │  ├──literal: +=
          │     │  └──whitespace: ␣
          │     ├──expression2:
          │     │  ├──digits: 432
          │     │  └──whitespace: ␣
          │     ├──literal: __ENDLINE__
          │     └──whitespace: ⏎␣␣␣␣
          ├──statement:
          │  └──assignment_stmt:
          │     ├──identifier:
          │     │  ├──identifier: fsdfs
          │     │  └──whitespace: ␣
          │     ├──assignment_operator:
          │     │  ├──literal: =
          │     │  └──whitespace: ␣
          │     ├──expression2:
          │     │  ├──digits: 411
          │     │  └──whitespace: ␣
          │     ├──literal: __ENDLINE__
          │     └──whitespace: ⏎␣␣␣␣␣⏎
          ├──literal: __DEDENT__
          └──whitespace: ␣⏎␣⏎

And the json (I haven't tested it, it seems like it's correct json)

    { "statement": [
        " \n",
        { "while_stmt": [
            "while",
            " ",
            { "expression2": [
                { "identifier": [
                    "var",
                ], },
            ], },
            ":",
            " \n",
            { "compound_stmt": [
                "__INDENT__",
                "\n    ",
                { "statement": [
                    { "jump_stmt": [
                        { "continue_stmt": [
                            "continue",
                            " ",
                            "__ENDLINE__",
                            "\n    ",
                        ], },
                    ], },
                ], },
                { "statement": [
                    { "expression_stmt": [
                        { "expression_dummy": [
                            "__EXPRESSION__",
                            " ",
                        ], },
                        "__ENDLINE__",
                        "\n    ",
                    ], },
                ], },
                { "statement": [
                    { "if_stmt": [
                        "if",
                        " ",
                        { "expression2": [
                            "1",
                        ], },
                        ":",
                        " \n    ",
                        { "compound_stmt": [
                            "__INDENT__",
                            "\n        ",
                            { "statement": [
                                { "assignment_stmt": [
                                    { "identifier": [
                                        "duh",
                                        " ",
                                    ], },
                                    { "assignment_operator": [
                                        "=",
                                        " ",
                                    ], },
                                    { "expression2": [
                                        { "paren_expr": [
                                            "(",
                                            { "expression2": [
                                                "43",
                                            ], },
                                            ",",
                                            " ",
                                            { "expression2": [
                                                { "identifier": [
                                                    "call",
                                                ], },
                                                { "paren_expr": [
                                                    "(",
                                                    { "expression2": [
                                                        "2",
                                                    ], },
                                                    ",",
                                                    " ",
                                                    { "expression2": [
                                                        "5",
                                                    ], },
                                                    ")",
                                                ], },
                                            ], },
                                            ")",
                                            " ",
                                        ], },
                                    ], },
                                    "__ENDLINE__",
                                    "\n        ",
                                ], },
                            ], },
                            { "statement": [
                                { "assignment_stmt": [
                                    { "identifier": [
                                        "ret",
                                        " ",
                                    ], },
                                    { "assignment_operator": [
                                        "=",
                                        " ",
                                    ], },
                                    { "expression2": [
                                        { "identifier": [
                                            "fun",
                                        ], },
                                        { "paren_expr": [
                                            "(",
                                            { "expression2": [
                                                "9",
                                            ], },
                                            ")",
                                            " ",
                                        ], },
                                    ], },
                                    "__ENDLINE__",
                                    "\n        ",
                                ], },
                            ], },
                            { "statement": [
                                { "assignment_stmt": [
                                    { "identifier": [
                                        "thing",
                                        " ",
                                    ], },
                                    { "assignment_operator": [
                                        "=",
                                        " ",
                                    ], },
                                    { "expression2": [
                                        { "identifier": [
                                            "stuff",
                                            " ",
                                        ], },
                                    ], },
                                    "__ENDLINE__",
                                    "\n         \n    ",
                                ], },
                            ], },
                            "__DEDENT__",
                            " \n    ",
                        ], },
                    ], },
                ], },
                { "statement": [
                    { "assignment_stmt": [
                        { "identifier": [
                            "thing",
                            " ",
                        ], },
                        { "assignment_operator": [
                            "=",
                            " ",
                        ], },
                        { "expression2": [
                            { "paren_expr": [
                                "(",
                                { "expression2": [
                                    "1",
                                ], },
                                ",",
                                " ",
                                { "expression2": [
                                    "34",
                                ], },
                                ")",
                                " ",
                            ], },
                        ], },
                        "__ENDLINE__",
                        "\n    ",
                    ], },
                ], },
                { "statement": [
                    { "assignment_stmt": [
                        { "identifier": [
                            "something",
                            " ",
                        ], },
                        { "assignment_operator": [
                            "+=",
                            " ",
                        ], },
                        { "expression2": [
                            "432",
                            " ",
                        ], },
                        "__ENDLINE__",
                        "\n    ",
                    ], },
                ], },
                { "statement": [
                    { "assignment_stmt": [
                        { "identifier": [
                            "fsdfs",
                            " ",
                        ], },
                        { "assignment_operator": [
                            "=",
                            " ",
                        ], },
                        { "expression2": [
                            "411",
                            " ",
                        ], },
                        "__ENDLINE__",
                        "\n     \n",
                    ], },
                ], },
                "__DEDENT__",
                " \n \n",
            ], },
        ], },
    ], },

My current goal is to iterate the parse tree and to generate C. I have litterally no idea how difficult that will be, but I'm still excited to try!

Don't hesitate to let me know what you think!

[foonathan]

Some comments:

• Any reason you're doing the preprocessing in python and not in C++?
• Since you don't change the whitespace rule in nested_expression, you don't need it and can recurse on expression directly.
• You don't appear to be using number, or xxx_literal?
• Why expression2 and not expression?
• If you don't care about the value, you should use dsl::digits and not dsl::integer. The latter also does an overflow check which you might not want or need.

[jokoon]

• Python is a bit simpler to use, but I could rewrite this part in C++, indeed. The slicing part is a bit tedious, but doable. I'm planning to still iterate the AST in python first, I was told that it's mostly the front end that's a bottleneck if I emit C.

• I removed that whitespace rule, did you mean to fix it like this?:

    struct expression : lexy::expression_production {
        static constexpr auto whitespace = dsl::ascii::space;

        struct nested_expr : lexy::transparent_production {
            static constexpr auto rule = dsl::recurse<struct expression>;
        };

        struct paren_expr{
            // corresponds to "paren_or_tuple" that was suggested
            static constexpr auto rule =
                dsl::parenthesized.list(dsl::p<nested_expr>, dsl::sep(dsl::comma));
        };

        struct expected_operand { static constexpr auto name = "expected operand"; };
        // We need to specify the atomic part of an expression.
        static constexpr auto atom = [] {
            // shouldn't use dsl::p<expression> instead dsl::p<nested_expr>
            auto var_or_call = dsl::p<identifier> >> dsl::if_(dsl::p<paren_expr>);
            return
                dsl::p<paren_expr>
                | var_or_call
                | dsl::p<string_literal>
                | dsl::p<number>
                | dsl::error<expected_operand>;
        }();

Or something else?

• I wanted to accept "0." and ".0" but if I remember it was not possible or too complicated, and I was oblivious to use the number rule in my expression rule. If I remember, it was easier to differentiate in the AST instead, so to answer, yes, I care about the value.

• I forgot to rename it to expression, I was lazy.

[jokoon]

I also need to fix how my C++ code generate some "jsonic" export, orphan commas are not allowed [1,2,3,].

I like C++, but I have to admit that my poor i5 is asthmatic and anything with lexy is not fast to build, although it might be because WSL2 has some virtualization cost or something.

[foonathan]

Or something else?

You can remove expressions::whitespace and expression::nested_expr entirely. It's only in the example because it changes the whitespace rules when there is a pending parenthesis.

so to answer, yes, I care about the value.

But you're using lexy::parse_as_tree which does not produce any values of rules.

anything with lexy is not fast to build

It is designed to be isolated to single translation units for that reason ;)

[jokoon]

So apparently I don't even need lexy::recurse... well okay!

But you're using lexy::parse_as_tree which does not produce any values of

rules. Oh now I understand, that's right that I could let lexy directly extract typed values instead ... I guess I would have to change my parser then.

…

On Tue, Sep 26, 2023 at 7:50 PM Jonathan Müller ***@***.***> wrote: Or something else? You can remove expressions::whitespace and expression::nested_expr entirely. It's only in the example because it changes the whitespace rules when there is a pending parenthesis. so to answer, yes, I care about the value. But you're using lexy::parse_as_tree which does not produce any values of rules. anything with lexy is not fast to build It is designed to be isolated to single translation units for that reason ;) — Reply to this email directly, view it on GitHub <#176 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAECK3LMBIWWIRRACXKB3ELX4MIU5ANCNFSM6AAAAAA5FETX4M> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jokoon]

Even if the is_last_node() is quite useful to generate json, that json becomes invalid if "noise" is removed (things like whitespace etc), because if a noise token is the last leaf, it will cause a faulty comma to be inserted in the json, unless I check the previous leaf and remove it as well, but that doesn't seem easy.

That leads me to think I should use lexy to generate C code, not just to emit a parse tree. I want to use python since it feels more comfortable to make a proof of concept.

Looking at parser_tree object, it doesn't seem I could really iterate over it and remove node/leaves I want, to have the "actual" is_last_node() I want after that. I don't really know if I could rewrite the parser_tree while filtering what I don't want.

[jokoon]

I fixed it by just iterating through the whole tree and filtering things I did not want.

Current todo list:

• else statements
• functions
• list and dict literals
• struples (as in tuple that gets translated to structs)
• picking C libraries that implement list and an hash tree

I will update it there:

https://github.com/jokoon/clak

[jokoon]

I wrote a longer article

https://jokoon.github.io/How_to_parse_a_C_like_language_with_lexy.html

It deals with parsing certain things like tuples, floating points/integers, indent/dedent tokens, and links to a parse tree walker.