[NUTCH-562] Port mime type framework to use Tika mime detection framework

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0

Environment

Mac Book Pro, Intel Core Duo 2.0 Ghz, 2.0 GB RAM, Mac OS X 10.4 although improvement is indep of env

• Assignee: Chris A. Mattmann
• Reporter: Chris A. Mattmann
• Due: 30/Sep/07
• Created: 29/Sep/07 04:36
• Updated: 08/Jun/11 21:35
• Resolved: 09/Oct/07 00:24

Description

With Tika (http://incubator.apache.org/tika/) nearing a stable 0.1 release candidate, I think it would be a good time to patch Nutch to use Tika's mime detection system (an improvement over the existing Nutch one written primarily by Jerome). Tika's mime system is based on the mime system from Freedesktop.org and includes several improvements over the existing Nutch mime system such as:

1. reliable XML-based content detection (a clear issue plaguing Nutch for some time now), ability to delineate between RSS, XML, ATOM, etc.
2. mime magic pattern matching, including support for multiple patterns
3. glob pattern matches (ability to support > 1)

I'll get together a patch and then attach it to the list once it's relatively stable.

Issue Links

• relates to: NUTCH-185 XMLParser is configurable xml parser plugin.