Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text missing or displaced in parsed table #540

Closed
pbonito opened this issue Dec 6, 2024 · 5 comments
Closed

Text missing or displaced in parsed table #540

pbonito opened this issue Dec 6, 2024 · 5 comments
Assignees
Labels
bug Something isn't working

Comments

@pbonito
Copy link

pbonito commented Dec 6, 2024

Bug

Text in table not present or displaced in parsed document

Steps to reproduce

Parse this pdf
parser_test.pdf

Screenshot 2024-12-06 at 18 57 31 Screenshot 2024-12-06 at 18 58 59

Docling version

Docling version: 2.8.1
Docling Core version: 2.5.1
Docling IBM Models version: 2.0.6
Docling Parse version: 2.1.2

Python version

Python 3.11.5

@pbonito pbonito added the bug Something isn't working label Dec 6, 2024
@maxmnemonic maxmnemonic self-assigned this Dec 9, 2024
@maxmnemonic
Copy link
Contributor

(accidentally deleted previous comment 😮‍💨)
@pbonito , please try this conversion with accurate setting for Tableformer, while not perfect it produces better result on your document.

In Docling CLI:
docling parser_test.pdf --to html --table-mode accurate

In Python:

from docling.datamodel.pipeline_options import PdfPipelineOptions, TableFormerMode
...

pipeline_options = PdfPipelineOptions(do_table_structure=True, table_structure_options=TableFormerMode.ACCURATE)
doc_converter = (
    DocumentConverter(  # all of the below is optional, has internal defaults.
        format_options={
            InputFormat.PDF: PdfFormatOption(
                pipeline_options=pipeline_options,
                pipeline_cls=StandardPdfPipeline,
                backend=DoclingParseDocumentBackend,
            )
        },
    )
)

This is the result (better than with fast model):
image

However, such tables are out of training distribution (hence mistakes) this we want to address with additional training data in the future.

@pbonito
Copy link
Author

pbonito commented Dec 10, 2024

@maxmnemonic I think it is
pipeline_options.table_structure_options.mode = TableFormerMode.ACCURATE
As you pointed out results improve but it is still missing one sentence. We can do more tests.
Can you share more on the strategy to improve results in future. Any plans?

@maxmnemonic
Copy link
Contributor

@pbonito, sure let me share some light on how it works, Tableformer is our model that we use in Docling to do table structure recognition. It's a single encoder / dual decoder model that trained to predict structural tags together with bounding boxes of the content. Docling then extracts text from a given bounding boxes and places it in the appropriate place in the structure. And as we see here those bounding boxes most likely fell short sometimes for such tables.

Thing is we trained Tableformer on public datasets such as FinTabNet and PubTab1M (and some others), these come from scientific papers, public financial reports, etc. Tables presented in such datasets while vary and enable model to do fairly good generalization, they miss some types of the tables, like the one we are looking at here, where there is a lot of text in each cell. Model was not used to see so much "volume" in each cell, and accuracy of predicted "content bounding box" drops.

Our current strategy is to fine tune model on the dataset that has such large-text tables in abundance, and in fact our team works on such dataset as we speak, so once we have dataset and do the fine-tuning we will just push new model weights that hopefully should improve the situation that we see here.

Hope this helps!

@itsainii
Copy link
Contributor

@maxmnemonic Is there an estimated timeline for when the new model weights will be pushed? I'm just curious about the expected update.

@maxmnemonic
Copy link
Contributor

@itsainii don't want to give timelines at the moment, before we test and prove the improvements. But I hope very early next year.

@DS4SD DS4SD locked and limited conversation to collaborators Dec 18, 2024
@cau-git cau-git converted this issue into discussion #621 Dec 18, 2024

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants