[TIKA-1302] Let's run Tika against a large batch of docs nightly

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: cli, general, server
• Assignee: Unassigned
• Reporter: Tim Allison
• Created: 16/May/14 19:55
• Updated: 29/Oct/19 15:52

Description

Many thanks to Lewis John McGibbney for TIKA-1301! Once we get nightly builds up and running again, it might be fun to run Tika regularly against a large set of docs and report metrics.

One excellent candidate corpus is govdocs1: http://digitalcorpora.org/corpora/files.

Any other candidate corpora?
William Palmer, have anything handy you'd like to contribute?
http://www.openplanetsfoundation.org/blogs/2014-03-21-tika-ride-characterising-web-content-nanite

Sub-Tasks

• Add robust tika-batch code: Resolved
• Find/configure a vm and gather initial corpus: Resolved
• Create tika-eval module: Resolved
• Create cron job to pull fresh versions of Tika: Open
• Add presentation layer for results of each run: Open
• Build simple stacktrace search interface: Open