Support optional stopword list

[cneud]

A common use case for OCR evaluation (e.g. for search engine indexing, text- and data mining, asf.) is to omit stopwords from the word evaluation to get an understanding of the correctness of "significant words" only.

It would therefore be useful if dinglehopper would also support the optional use of a stopword list provided via parameter/config file. This is already supported in ocrevalUAtion.

[mikegerber]

Given

GT: the quick brown fox jumps over the lazy dog
OCR: the quick brown fox jumps over they lazer dog

with a stop word list:

the

would only count 1 error (lazy vs lazer).

[cneud]

Just leaving this here for documentation - this is also sometimes referred to as "significant words" evaluation.

The number of occurrences of content words for which users might be interested in searching, excluding stop-listed words, such as "the", "he", "it", etc.
Measuring Mass Text Digitization Quality and Usefulness
Measuring the OCR Accuracy across The British Library’s Newspaper Archive

[cneud]

would only count 1 error (lazy vs lazer).

Exactly. Any words appearing in the GT and also in the stopword list are ignored when computing the "significant words" accuracy rate.