Kensho Wikimedia for Natural Language Processing - Preprocessor

kwnlp_preprocessor is a Python package to help you convert raw Wikimedia data to standard formats.

Quick Install (Requires Python >= 3.6)

# Install the pre-commit setup (linters in our case)
pip install pre-commit
pre-commit install

pip install . # This package is not on pypi yet
# or "pip install -e ." to install in editable mode

Status

This code is not battle tested production code. It is mostly used by the R&D team to prototype new ideas using Wikimedia data.