How to parse a string without quotes and with code_point?

[Life4gal]

lexy/examples/json.cpp

Lines 209 to 260 in 5e2601b

	struct string : lexy::token_production
	{
	struct invalid_char
	{
	static LEXY_CONSTEVAL auto name()
	{
	return "invalid character in string literal";
	}
	};
	// A mapping of the simple escape sequences to their replacement values.
	static constexpr auto escaped_symbols = lexy::symbol_table<char> //
	.map<'"'>('"')
	.map<'\\'>('\\')
	.map<'/'>('/')
	.map<'b'>('\b')
	.map<'f'>('\f')
	.map<'n'>('\n')
	.map<'r'>('\r')
	.map<'t'>('\t');
	// In JSON, a Unicode code point can be specified by its encoding in UTF-16:
	// * code points <= 0xFFFF are specified using `\uNNNN`.
	// * other code points are specified by two surrogate UTF-16 sequences.
	// However, we don't combine the two surrogates into the final code point,
	// instead keep them separate and require a later pass that merges them if necessary.
	// (This behavior is allowed by the standard).
	struct code_point_id
	{
	// We parse the integer value of a UTF-16 code unit.
	static constexpr auto rule
	= LEXY_LIT("u") >> dsl::code_unit_id<lexy::utf16_encoding, 4>;
	// And convert it into a code point, which might be a surrogate.
	static constexpr auto value = lexy::construct<lexy::code_point>;
	};
	static constexpr auto rule = [] {
	// Everything is allowed inside a string except for control characters.
	auto code_point = (-dsl::unicode::control).error<invalid_char>;
	// Escape sequences start with a backlash and either map one of the symbols,
	// or a Unicode code point.
	auto escape
	= dsl::backslash_escape.symbol<escaped_symbols>().rule(dsl::p<code_point_id>);
	// String of code_point with specified escape sequences, surrounded by ".
	// We abort string parsing if we see a newline to handle missing closing ".
	return dsl::quoted.limit(dsl::ascii::newline)(code_point, escape);
	}();
	static constexpr auto value = lexy::as_string<ast::json_string, lexy::utf8_encoding>;
	};

The above example shows how to parse a string with code_point and surrounded by double quotes, but it can't be used with strings that aren't surrounded by anything.

In this case, lexy::dsl::identifier is generally used as an alternative to parsing strings, but there is a problem in that lexy::dsl::identifier only parses raw strings, and does not handle code_points.

I think the solution could be to first get a lexeme with lexy::dsl::identifier and then parse the lexeme. But I'm not sure how to do that.

[foonathan]

If the string doesn't have quotes, what terminates the string? You can pass those terminators to dsl::delimited: https://lexy.foonathan.net/reference/dsl/delimited/#delimited

[foonathan]

You shouldn't need dsl::peek before dsl::delimited, that is already the behavior it has on its own. If you don't want to consume the newline, you can use dsl::peek(dsl::unicode::newline) as the closing terminator.

[Life4gal]

constexpr static auto rule = dsl::delimited(dsl::peek(dsl::unicode::print - dsl::unicode::newline - dsl::unicode::blank), dsl::peek(dsl::unicode::newline))(code_point, escape) | dsl::error<invalid_string>;

constexpr static auto value = lexy::as_string<std::string>;

The above rule solves almost all problems, but it allows empty strings (instead of generating an error), how to reject this?

[Life4gal]

https://godbolt.org/z/x56TosjGj

[foonathan]

It doesn't allow empty strings? https://lexy.foonathan.net/playground/?id=1rasdbjY3&mode=tree

[Life4gal]

https://lexy.foonathan.net/playground/?id=foY5n6s4W&mode=tree

That's what I need! Although I need to manually remove the extra whitespace on the string right side in the callback (I want to allow whitespace inside the string after all)

[Life4gal]

It doesn't seem to be fully resolved, it looks fine in the playground.

https://lexy.foonathan.net/playground/?id=aseo8xYGP&mode=tree

But:

Why is tab considered an illegal character? (BTW, I was using -control before)

[Life4gal]

:(

[foonathan]

You said that your delimited should only contain printable Unicode characters, and TAB just isn't considered to be one according to the Unicode standard. So you'd want dsl::unicode::print / dsl::lit_c<'\t'>.

(The playground translates tab to four spaces, which is why it works there.)