Nutch
  1. Nutch
  2. NUTCH-562

Port mime type framework to use Tika mime detection framework

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Mac Book Pro, Intel Core Duo 2.0 Ghz, 2.0 GB RAM, Mac OS X 10.4 although improvement is indep of env

      Description

      With Tika (http://incubator.apache.org/tika/) nearing a stable 0.1 release candidate, I think it would be a good time to patch Nutch to use Tika's mime detection system (an improvement over the existing Nutch one written primarily by Jerome). Tika's mime system is based on the mime system from Freedesktop.org and includes several improvements over the existing Nutch mime system such as:

      1. reliable XML-based content detection (a clear issue plaguing Nutch for some time now), ability to delineate between RSS, XML, ATOM, etc.
      2. mime magic pattern matching, including support for multiple patterns
      3. glob pattern matches (ability to support > 1)

      I'll get together a patch and then attach it to the list once it's relatively stable.

      1. tika-0.1-dev.jar
        105 kB
        Chris A. Mattmann
      2. NUTCH-562.Mattmann.patch.txt
        309 kB
        Chris A. Mattmann

        Issue Links

          Activity

          Chris A. Mattmann created issue -
          Chris A. Mattmann made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          Chris A. Mattmann made changes -
          Link This issue relates to NUTCH-185 [ NUTCH-185 ]
          Chris A. Mattmann made changes -
          Attachment NUTCH-562.Mattmann.patch.txt [ 12367214 ]
          Chris A. Mattmann made changes -
          Attachment tika-0.1-dev.jar [ 12367215 ]
          Chris A. Mattmann made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Chris A. Mattmann made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Julien Nioche made changes -
          Component/s mime_type_detector [ 12311982 ]

            People

            • Assignee:
              Chris A. Mattmann
              Reporter:
              Chris A. Mattmann
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development