Extend Top Level Section detection to work for more 10-Q reports

[Elijas]

Note
This is an "umbrella" issue that groups several related issues into a single GitHub Issue. This approach helps us keep the Issues list organized and less cluttered.

What are "Top Level Sections"?

Top-level sections are needed in most 10-Q reports as required by the SEC. These sections are:

part1:
- part1item1 - Financial Statements
- part1item2 - Discussion from Management about Financial Situation and Operations
- part1item3 - Details about Market Risk
- part1item4 - Controls and Procedures

part2:
- part2item1 - Legal Problems
- part2item1A - Risk Factors
- part2item2 - Sales of Equity Securities without Registering and Use of the Money
- part2item3 - Defaults on Senior Securities
- part2item4 - Safety in Mines
- part2item5 - Other Information
- part2item6 - Exhibits

Introduction to Testing

We have set up generalization tests to test how certain features of our parser generalize across our document dataset.

Execute this command to run the tests:

task generalization-tests

You should see an output similar to this:

In this image, we see that 42 reports have their top-level sections identified correctly, while 30 reports have some issues.

Enabling a failing test

In the YAML file, uncomment (i.e. remove the # symbol) from some report, to include it in the test. For example, to include "10-Q ABT Abbott Laboratories", uncomment the line in:

https://github.com/alphanome-ai/sec-parser/blob/3078b5fd6e24f0dae796a76217bff25204b356af/tests/generalization/processing_steps/test_top_level_section_title_classifier.yaml#L4-L6

so it becomes:

  - 0001551152-23-000033 # 10-Q ABBV AbbVie Inc.
  - 0001628280-23-027203 # 10-Q ABT Abbott Laboratories
  - 0000796343-23-000205 # 10-Q ADBE Adobe Inc.

Now, executing the test command:

task generalization-tests

again would include the failing test:

Helpful tools

You can use the UI dashboard to check the elements

task launch-debug-dashboard

Consider viewing the source code to better understand any problems. Go to "Show Contents" -> "HTML Code"

[Elijas]

Note
Moved from: alphanome-ai/sec-parser#31
Original author: @INF800

AssertionError: Missing: ['part2item6'] is raised in generalization tests if exhibits appear with a table at the same level as top-level section title.

This is a common issue in:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_RTX_0000101829-23-000032] - AssertionError: Missing: ['part2item6']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_TMO_0000097745-23-000059] - AssertionError: Missing: ['part2item6']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_UPS_0001090727-23-000038] - AssertionError: Missing: ['part2item6']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MCD_0000063908-23-000076] - AssertionError: Missing: ['part2item6']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MDT_0001613103-23-000128] - AssertionError: Missing: ['part2item6']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_GM_0001467858-23-000098] - AssertionError: Missing: ['part2item6'], Unexpected: ['part1item1']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_KHC_0001637459-23-000114] - AssertionError: Missing: ['part2item6']

Almost all of them (RTX, TMO, MDT, KHC, GM) have the following pattern:

<div ...>

    <span ...>
    Item 6. Exhibits
    </span>

    <table ...>
    ...
    </table>

</div>

UPS with small difference:

<div ...>

    <span ...>
    Item 6. 
    </span>
    <span ...>
    Exhibits
    </span>

    <table ...>
    ...
    </table>

</div>

MCD has major difference where the title "Item 6. Exhibits" is written inside the table

<div>
    <table>
        <tr>
        <tr/>

        <tr>
            <td ...>
                <div ...>        
                    <span ...>
                        Item 6. Exhibits
                    </span>
                </div>
            </td>
        </tr>

        ...

        <tr>
        <tr/>
        <tr>
        <tr/>
    </table>
</div>

Screenshots for two major cases:

Table at the same level as top-level section title

Top-level section title written inside table:

[Elijas]

Note
Moved from: alphanome-ai/sec-parser#30
Original author: @INF800

Currently, TextElement can become HighlightedTextElement only if 80% of the text content is bold with some font weight. This is not an ideal scenario in cases such as part2item3 of 10-Q_MMM_0000066740-23-000058 which looks like:

where ~55% of text content is bold with some font weight. The style_string default dict for it looks like:

Because of this TextElement cannot become HighlightedTextElement which in turn cannot become TitleElement which in turn cannot become TopLevelSectionElement.

Solution:

Change PERCENTAGE_THRESHOLD from 80 to a value less than 54.9.

https://github.com/alphanome-ai/sec-parser/blob/58304f04f809c00b9b613028a1a961044259556d/sec_parser/semantic_elements/highlighted_text_element.py#L60

In order to find a sweet spot I checked if there were more fails like these where the threshold needs to be decreased, but could not find any in the dataset. So, I think the best value as of now would be 50 percent assuming neither the trailing description or the bold heading has more text content.

[Elijas]

Note
Moved from: A comment from alphanome-ai/sec-parser#30
Original author: @INF800

Found a case where un-highlighted text can be a title too. (See Item 1)

Ticker - RKT

[INF800]

The discussion from Dicord: click here

There are also cases where top-level section headings cannot be caught using regex. For eg there is no "item2." suffix before heading or the title itself is different

Case 1: INTC

A. "Consolidated Condensed Statements of Income" instead of traditional "Item 1. Financial Statements."

B. No "Item 3." in "Quantitative and Qualitative Disclosures About Market Risk"

C. This example does not have the "Part I - Financial Information" heading as text as well. (Also the case in 10-Q_MARA_0001493152-23-027174)

D. It contains the "Form 10-Q Cross-Reference Index" section with hyperlinks to actual sections.

Test:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_INTC_0000050863-23-000069] - AssertionError: No top level sections found

Case 2: GE

Similar to case 1, "FORM 10-Q CROSS REFERENCE INDEX" section is present, and there are no prefixes to section headings.

Test:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_GE_0000040545-23-000198] - AssertionError: No top level sections found

Case 3: JPM

The "Item 2. Management’s Discussion and Analysis of Financial Condition and Results of Operations" (as in it's table of contents) is present in its section without any heading.

Test:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_JPM_0000019617-23-000432] - AssertionError: 10-Q document should have part1 and part2, but has []

Case 4: MS

All top-level sections are present without item-{id} prefix. Some example screenshots:

Test:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MS_0000895421-23-000374] - AssertionError: No top level sections found

Case 5: HON

Missing "ITEM 1." prefix

Test:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_HON_0000773840-23-000073] - AssertionError: Missing: ['part1item1']

---

---

Some Other Off-Topic Fails

1. Filing: 10-Q_F_0000037996-23-000050:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_F_0000037996-23-000050] - AssertionError: Missing: ['part1item3']

2. Filing: 10-Q_SCHW_0000316709-23-000037:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_SCHW_0000316709-23-000037] - AssertionError: Missing: ['part1item1']

part1item1 present after item 2 and 3

3. Filing 10-Q_MULN_0001558370-23-014900:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MULN_0001558370-23-014900] - AssertionError: Missing: ['part1item3', 'part1item4']

Text Element could not be divided, as a result part 3 and 4 were missed.

The browser tab was hanging when tried to extract HTML code using the demo app.

4. Filing 10-Q_CL_0000021665-23-000031:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_CL_0000021665-23-000031] - AssertionError: Missing: ['part1item1']

Missing item 1 heading altogether after part 1 heading.

5. Filing 10-Q_MMM_0000066740-23-000058:

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MMM_0000066740-23-000058] - AssertionError: Missing: ['part2item3']

Could not split text text element to incorporate item 3.

The browser tab was hanging when tried to extract HTML code using the demo app.

[Elijas]

AA/000119312518236766

[Elijas]

Current state with sec-parser==0.40.0: 53 passing, 20 failing (73% success rate)

FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_VZ_0000732712-23-000055] - AssertionError: Missing: ['part2item5']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_INTC_0000050863-23-000069] - AssertionError: No top level sections found
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MA_0001141391-23-000131] - AssertionError: Missing all expected 10-Q "part" sections, Unexpected: ['part1item1', 'part1item2', 'part1item3', 'part1item4'...
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MS_0000895421-23-000374] - AssertionError: No top level sections found
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_AMD_0000002488-23-000139] - AssertionError: Missing: ['part2item5']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_T_0000732717-23-000047] - AssertionError: Unexpected: ['part2item2', 'part2item6']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_BAC_0000070858-23-000176] - AssertionError: Missing: ['part1item1', 'part1item2', 'part1item3', 'part1item4', 'part2item1', 'part2item1a'], Unexpected: ['...
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_ABT_0001628280-23-027203] - assert 1 == 0
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_DOW_0001751788-23-000128] - AssertionError: 10-Q document should have part1 and part2, but has ['part1']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MCD_0000063908-23-000076] - AssertionError: Missing: ['part2item6']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_HON_0000773840-23-000073] - AssertionError: Missing: ['part1item1']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_WBA_0001618921-23-000043] - AssertionError: Missing: ['part1item1', 'part1item2', 'part1item3', 'part1item4', 'part2item1a', 'part2item2', 'part2item5']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_GE_0000040545-23-000198] - AssertionError: No top level sections found
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_TRV_0000086312-23-000063] - AssertionError: 10-Q document should have part1 and part2, but has ['part2']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_GM_0001467858-23-000098] - AssertionError: Unexpected: ['part1item1']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_CL_0000021665-23-000031] - AssertionError: Missing: ['part1item1']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_AXP_0000004962-23-000038] - AssertionError: Missing: ['part1item1']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_MARA_0001493152-23-027174] - AssertionError: 10-Q document should have part1 and part2, but has ['part2']
FAILED tests/generalization/processing_steps/test_top_level_section_title_classifier.py::test_top_level_section_title_classifier[10-Q_SCHW_0000316709-23-000037] - AssertionError: Missing: ['part1item1']

[INF800]

Caution

Note: There are cases where even sec-api.io fails to extract top level sections correctly. More about it with examples is present in this discord thread.