[lexical-markdown] Fix: normalize markdown in $convertFromMarkdownString to comply with CommonMark spec

[GermanJablo]

Problem

In markdown, the concept of "empty paragraphs" does not exist.
Blocks must be separated by an empty line, and non-empty adjacent lines must be merged.

Let's take this markdown as an example:

one
two

three

Currently, it is converted to the following (incorrect):

<p>one<br>two</p>
<p>three</p>

When in fact, the correct output should be this (proof):

<p>onetwo</p>
<p>three</p>

For context, we are trying to import real mdx files into Lexical, and because of this issue, the current output contains some errors, such as this one here.

Solution

At first I thought it would be enough to remove all single line breaks (\n), which were not accompanied by further consecutive line breaks (\n\n).

However, the solution was not that easy, as there were some tricky edgecases. To mention just a couple of examples:

• within a code block, no line breaks should be removed (ref).
• consecutive heading + paragraph should not be combined, but list-item + "paragraph" should (ref).

That's why I wrote a function called sanitizeMarkdown to cover these and all the other cases I found, and which is now run inside $convertFromMarkdownString.

A few tests required changes. Since the correct result was not obvious, I left some comments linking to permalinks in the CommonMark playground.

Future work

If one would like to add a hard line break, there are 3 ways to do so in markdown (source):

1. an html <br> tag,
2. ending the line with two spaces,
3. or ending the line with \.

Right now, the only option that works with Lexical is the first one. I've left a TO-DO comment indicating that it would be nice to support the other 2 in the future.

Test plan

Before

Screen.Recording.2024-09-08.at.12.45.19.AM.mov

After

Screen.Recording.2024-09-08.at.12.43.14.AM.mov

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
lexical	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 9, 2024 1:31pm
lexical-playground	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 9, 2024 1:31pm

[github-actions]

size-limit report 📦

Path	Size
lexical - cjs	29.61 KB (0%)
lexical - esm	29.49 KB (0%)
@lexical/rich-text - cjs	38.07 KB (0%)
@lexical/rich-text - esm	31.3 KB (0%)
@lexical/plain-text - cjs	36.72 KB (0%)
@lexical/plain-text - esm	28.65 KB (0%)
@lexical/react - cjs	39.89 KB (0%)
@lexical/react - esm	32.79 KB (0%)

[etrepum]

Approving because it does seem like an incremental improvement but I think there's still other classes of edge cases in here

[GermanJablo]

Thanks for your comments @etrepum, I already solved them

[etrepum]

The lack of spec compliance makes it hard to review thoroughly since it's easy to find things that don't work correctly, but it does pass our tests and I think this particular change is unlikely to cause new problems.

[potatowagon]

The lack of spec compliance makes it hard to review thoroughly since it's easy to find things that don't work correctly, but it does pass our tests and I think this particular change is unlikely to cause new problems.

thankyou for your review