quipucamayoc is a Python package that simplifies the extraction of historical data from scanned images and PDFs. It's designed to be modular and so it can be used together with other existing tools, and can be extended easily by users.
For an overview of how to use quipucamayoc to digitize historical data, see this research article, which amongst other things details the different steps involved, the methods used, and provides practical examples.
For an user guide, documentation, and installation instructions, see http://scorreia.com/software/quipucamayoc/ (TODO).
If you want to contribute by improving the code or extending its functionality (much welcome!), head here.
To manage quipucamayoc using pip, open the command line and run:
pip install quipucamayocto installpip install quipucamayoc[dev]to include extra dependencies used when developing the code
pip install -U quipucamayocto upgradepip uninstall quipucamayocto remove
Note that quipucamayoc has been tested against Python 3.10 and newer versions, but should also work with Python 3.9.
After cloning the repo to your computer and navigating to the quipucamayoc folder, run:
pip install .to install the package locallypip install --no-cache-dir --editable .[dev]to install locally with a symlink so changes are automatically updated (recommended for developers)
quipucamayoc can use AWS's Textract to OCR text and tables. Its configuration is quite cumbersome, so it has been automated for you. To do so, first install quipucamayoc and then follow these steps to specify your credentials:
- Ensure you have an Amazon AWS account.
- AWS Textract prices are listed here. If you just created an account you should be able to use the free tier. Otherwise, you might need to set up a payment method.
- Now you need to create credentials so you can access AWS programmatically.
- The simplest method is to go to the security credentials page (you can also go to it from the AWS console: click on your name on the top-right > click on security credentials). Then, ignore the security warning (see below), go the next page, scroll to Access Keys and click create an access key. Copy the
Access keyandSecret access keystrings (akin to username and passwords). - (TODO) Alternatively, AWS now recommends an alternative: instead of creating access keys for your user, create a new user with a more restricted access and then create an access key for this user. In this way, if you somehow lose your credentials (e.g. your computer is hacked) then hackers are limited in what they can do. To create an user, go to the Identity and Access Management (IAM) console, and create a new user: Access Management > Users > Add Users. Select a name, press Next, then you need to attach certain policies (TODO: find out which policies are needed).
- Download and install the
awscommand line interface (CLI). Update:quipucamayocinstalls theawsclipackage so this step might not be necessary anymore. - Go to the command line, and type
aws configureto enter your credentials. You need to enter yourAccess keyandSecret Access Keystrings. TheDefault region namecan be the AWS region of your preference (most likely,aws-east-1) or left empty.Default output formatcan be left empty. - From the command line, you can now run the quipucamayoc command
quipu aws install. This will setup your AWS account so you can useTextractprogrammatically (i.e., create an S3 bucket, an SNS topic, a SQS queue, a user with the required credentials, and then configure the user, bucket, topic, and queue so they can talk to each other.)
Notes:
- You can avoid steps 3-4 by directly [writing your credentials[(https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)] to the
credentialsfile. - If you want to remove all quipucamayoc artifacts from your AWS account, you can run
quipu aws uninstallfrom the command line.
- From the command line, you can extract tables using AWS via
quipu extract-tables --filename <myfile.pdf>
- Automatically set up Textract pipeline
- Expose key functions as command line tools
- Allow parallel (async?) tasks. Useful for OpenCV (CPU-intensive) and Textract calls (IO-intensive). Consider also uvloop
- Include Poppler by default on Windows
- Add mypy/(flake8|black)
- https://github.com/qurator-spk/eynollah
- https://github.com/qurator-spk/sbb_binarization
- https://github.com/leonlulu/DeepLayout
Feel free to submit push requests. For consistency, code should comply with pep8 (as long as its reasonable), and with the style guides by @kennethreitz and google. Read more here.
- Sergio Correia, Stephan Luck: “Digitizing Historical Balance Sheet Data: A Practitioner's Guide”, 2022; arXiv:2204.00052.
@misc{quipucamayoc,
Author = {Correia, Sergio and Luck, Stephan},
Title = {Digitizing Historical Balance Sheet Data: A Practitioner's Guide},
Year = {2022},
eprint = {arXiv:2204.00052},
journal={arXiv preprint arXiv:2204.00052}
}Quipucamayoc is built upon the work and improvements of many users and developers, from which it was heavily inspired, such as:
It is also relies for most of its work on the following open source projects:
Quipucamayoc is developed under the GNU Affero GPL v3 license.
The quipucamayocs were the Inca empire officials in charge of desciphering (amonst other things) accounting information stored in quipus. Our goal for this package is to act as a sort of quipucamayoc, helping researchers in desciphering and extracting historical information, particularly balance sheets and numerical records.
