Refactoring all PDF loader and parser

[pprados]

WIP

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	❌ Failed (Inspect)			Dec 16, 2024 5:28pm

[vercel]

Deployment failed with the following error:

The provided GitHub repository does not contain the requested branch or commit reference. Please ensure the repository is not empty.

[efriis]

Hey @pprados! I don't think this work is going to result in a PR that is reviewable if it's only partially done and already adding 7000 lines.

What is your goal in this work?

[pprados]

@efriis

I'm well aware of that. That's why a meeting is to be organized with LangChain (via AXA France), normally next week, to see how best to proceed, with @eyurtsev.

We're sorry, it may take you several hours to validate it. The changes are important and cannot be published one after the other, as everything is linked. It's going to be difficult to cut the code into 12 successive PRs, and end up with the same result. And that's going to take months. All this work is validated by two matrix tests, ensuring the consistency of all modifications.

In order to qualify all the code, we worked on a separate project, using the langchain-common structure. In this way, we can compare the results of the historical implementation with the new ones.

We understand that it's important to ensure that changes don't have a significant impact on existing code. That's why we used a parallel project, using the langchain-common structure, to test PDF readings before and after modifications. This allows us to compare results. You'll find all the files here. The only difference is the name to import classes.

We prepare the PR and its description. Look here to understand our work. We welcome any suggestions you may have to help us integrate it.

You can now pre-view the description. The final version won't be far off.

The aim is to submit the PR in early 2025.

[efriis]

ah got it - is there an issue or discussion of proposed changes? It might be easier to discuss ideas than these code changes

[pprados]

@efriis
90% of my customers work with PDF files and don't have a satisfactory solution at the moment. They cobble together solutions outside langchain (pdf processing outside loaders/parsers), sacrificing a good part of the benefits of this framework. Seeing the same problems over and over again, and the same bad solutions, I couldn't let them go on like that. I had to deal with the problem in the best possible way, for my customers and all LangChain users.

I've been funded by my client to simultaneously help projects in Belgium, Switzerland, Spain, Italy and France. I couldn't wait for a discussion on the subject. In the end, what I'm proposing is a no-brainer:

• Integrate tables where possible
• to indicate what you want to do with the images (invoke a multimodal LLM, for example)
• standardize the various solutions to eventually enable automatic selection of the parser according to document characteristics

[efriis]

will let you and eugene discuss in the time you have scheduled.