Skip to content
/ dedoc Public

Dedoc is a library (service) for automate documents parsing and bringing to a uniform format. It automatically extracts content, logical structure, tables, and meta information from textual electronic documents. (Parse document; Document content extraction; Logical structure extraction; PDF parser; Scanned document parser; DOCX parser; HTML parser

License

Notifications You must be signed in to change notification settings

ispras/dedoc

Repository files navigation

Dedoc

Telegram image GitHub release PyPI version PyPI downloads Docker Hub License Demo dedoc-readme.hf.space Documentation Status CI tests

Dedoc

Dedoc is an open universal system for converting documents to a unified output format. It extracts a document’s logical structure and content: tables, text formatting and metadata. The document’s content is represented as a tree storing headings and lists of any level. Dedoc can be integrated in a document contents and structure analysis system as a separate module.

Star History

Star History Chart

Workflow

Workflow

Workflow description is given here

Features and advantages

Dedoc is implemented in Python and works with semi-structured data formats (DOC/DOCX, ODT, XLS/XLSX, CSV, TXT, JSON) and unstructured data formats like images (PNG, JPG etc.), archives (ZIP, RAR etc.), PDF and HTML formats. Document structure extraction is fully automatic regardless of input data type. Metadata and text formatting are also extracted automatically.

In 2022, the system won a grant to support the development of promising AI projects from the Innovation Assistance Foundation (Фонд содействия инновациям).

Dedoc provides:

  • Extensibility due to flexible addition of new document formats and easy change of an output data format.
  • Support for extracting document structure out of nested documents having different formats.
  • Extracting various text formatting features (indentation, font type, size, style etc.).
  • Working with documents of various origin (statements of work, legal documents, technical reports, scientific papers) allowing flexible tuning for new domains.
  • Working with PDF documents containing a textual layer:
    • Support to automatically determine the correctness of the textual layer in PDF documents;
    • Extract containing and formatting from PDF-documents with a textual layer using the developed interpreter of the virtual stack machine for printing graphics according to the format specification.
  • Extracting table data from DOC/DOCX, PDF, HTML, CSV and image formats:
    • Recognizing a physical structure and a cell text for complex multipage tables having explicit borders with the help of contour analysis.
  • Working with scanned documents (image formats and PDF without text layer):
    • Using Tesseract, an actively developed OCR engine from Google, together with image preprocessing methods.
    • Utilizing modern machine learning approaches for detecting a document orientation, detecting single/multicolumn document page, detecting bold text and extracting hierarchical structure based on the classification of features extracted from document images.

Document format description

The system processes different document formats. The main formats are listed below:

Format group Description
Office formats DOCX, XLSX, PPTX and formats that can be converted to them. Handling of these formats is held by analysis of format inner representation and using specialized libraries (python-docx, BeautifulSoup)
HTML, EML, MHTML HTML documents are parsed using tags analysis, HTML handler is used for handling documents of other formats in this group
TXT Only raw textual content is analyzed
Archives Attachments of the archive are analyzed
PDF, document images Copyable PDF documents (with a textual layer) can be handled using pdfminer-six library or tabby software. Non-copyable PDF documents or images are handled using Tesseract-OCR, machine learning methods (including neural network methods) and image processing methods

Examples of processed scanned documents

  • Dedoc can only process scanned black and white documents, such as technical specifications, regulations, articles, etc.

Document examples

  • In particular, dedoc recognizes tabular information only from tables with explicit boundaries. Here are examples of documents that can be processed by an dedoc's image handler:

Table parsing example

  • The system also automatically detects and corrects the orientation of scanned documents

Examples of structure extractors

Law structure example

Tz structure example

Impact

This project may be useful as a first step of an automatic document analysis pipeline (e.g. before the NLP part). Dedoc is in demand for information analytic systems, information leak monitoring systems, as well as for natural language processing systems. The library is intended for application use by developers of systems for automatic analysis and structuring of electronic documents, including for further search in electronic documents.

Documentation

Relevant documentation of dedoc is available here

Demo

Publications related to dedoc

Join Our Community

Have questions or want to discuss Dedoc? Join our Telegram chat and connect with the community and the developers.

Join our Telegram channel to get notifications about the most recent updates.

Installation instructions

This project has a REST api and you can run it in Docker container. Also, dedoc can be installed as a library via pip. There are two ways to install and run dedoc as a web application or a library that are described below.

Install and run dedoc using docker

You should have git and docker installed for running dedoc by this method. This method is more flexible because it doesn't depend on the operating system and other user's limitations, still, the docker application should be installed and configured properly.

If you don't need to change the application configuration, you may use the built docker image as well.

Work with dedoc as service

1. Pull the image

docker pull dedocproject/dedoc

2. Run the container

docker run -p 1231:1231 --rm dedocproject/dedoc python3 /dedoc_root/dedoc/main.py

Go to dockerhub to get more information about available dedoc images.

If you need to change some application settings, you may update config.py according to your needs and re-build the image. You can build and run image:

1. Clone the repository

git clone https://github.com/ispras/dedoc

2. Go to the dedoc directory

cd dedoc

3. Build the image and run the application

docker compose up --build

4. Run container with tests

test="true" docker compose up --build

If you need to change some application settings, you may update config.py according to your needs and re-build the image.

Install dedoc using pip

If you don't want to use docker for running the application, it's possible to run dedoc locally. However, it isn't suitable for any operating system (Ubuntu 20+ is recommended) and there may be not enough machine resources for its work. You should have python (python3.8, python3.9 are recommended) and pip installed. Installation instructions via pip are available here.

Install and run dedoc from sources

You can also install and run dedoc locally from sources using this instruction.

About

Dedoc is a library (service) for automate documents parsing and bringing to a uniform format. It automatically extracts content, logical structure, tables, and meta information from textual electronic documents. (Parse document; Document content extraction; Logical structure extraction; PDF parser; Scanned document parser; DOCX parser; HTML parser

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages