A python3 library to link a dirty CSV file with a clean reference table. It is meant to as generic as possible and includes a labeller to learn optimal parameters for each matching scenario.
This library relies on Elasticsearch. We used version 5.6.7 for developpment.. We recommend Elasticsearch 5.X. Instructions here.
pip3 install merge-machine
From source (recommended, for the meantime...):
git clone https://github.com/entrepreneur-interet-general/Merge-Machine.git
cd Merge-Machine
pip3 install -e .
See examples/example.py.
See examples/gen_resource_example.py.
See HOW_TO.md.
The reference is indexed in Elasticsearch with multiple analyzers (languages specific, integers, n_grams...). The labeller then proposes training samples from the source which it tries to match to rows of the reference file. Upon user confirmation (match / not match) it updates its belief on which Elasticsearch queries are most performant to use for matching. When labelling is over, the "best query" (a weighted combination of multiple ES queries with different analyzers on different fields) is used for each row of the source to try to find a match in the ES-indexed referential.
Feel free to report bugs via issues and make pull requests...
This library was developped by Léo Bouloc during 10 months in 2017 at the French Ministry of Research and Higher Education in the context of the "Entrepreneur d'Intérêt Général" program funded by the French Government.
This library was developped as a component of larger matching service:
- ONLINE SERVICE COMING SOON !!!
- code
Check out:
- The Merge Machine, A simple web interface to link your data
Other similar libraries include: