Nutch
  1. Nutch
  2. NUTCH-562

Port mime type framework to use Tika mime detection framework

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Mac Book Pro, Intel Core Duo 2.0 Ghz, 2.0 GB RAM, Mac OS X 10.4 although improvement is indep of env

      Description

      With Tika (http://incubator.apache.org/tika/) nearing a stable 0.1 release candidate, I think it would be a good time to patch Nutch to use Tika's mime detection system (an improvement over the existing Nutch one written primarily by Jerome). Tika's mime system is based on the mime system from Freedesktop.org and includes several improvements over the existing Nutch mime system such as:

      1. reliable XML-based content detection (a clear issue plaguing Nutch for some time now), ability to delineate between RSS, XML, ATOM, etc.
      2. mime magic pattern matching, including support for multiple patterns
      3. glob pattern matches (ability to support > 1)

      I'll get together a patch and then attach it to the list once it's relatively stable.

      1. NUTCH-562.Mattmann.patch.txt
        309 kB
        Chris A. Mattmann
      2. tika-0.1-dev.jar
        105 kB
        Chris A. Mattmann

        Issue Links

          Activity

          Hide
          Chris A. Mattmann added a comment -

          This will allow an XML parser to properly be called because the appropriate mime type is detected.

          Show
          Chris A. Mattmann added a comment - This will allow an XML parser to properly be called because the appropriate mime type is detected.
          Hide
          Chris A. Mattmann added a comment - - edited

          Initial patch for comments:

          1. This patch removes the MimeType system, and its associated java src files, config files and unit tests from Nutch. This information is in Tika now and is replaced by its TIka counterparts.
          2. This patch uses the unreleased 0.1-dev version of Tika. When 0.1 is officially released, we can convert to that, though I don't anticipate any MimeType API changes between now and then.
          3. All unit tests for core and plugins pass, however, it's probably a good idea to run at least a small crawl with this patch and see if everything works fine. I don't really have the time for this now, so anyone want to try? (cough cough Dogacan cough cough )
          4. It's worth noting that this MimeType system from Tika changes the traditional Nutch mime type system (IMO for the better) in a couple of ways. First, whereas the old MimeType system was very happy to return null in places where it couldn't figure out the MimeType, this system tries to return a "default" MimeType (which in this case is "application/octet-stream") if it can't guess the mime type from those that it knows about. Second, this mime type system uses a different type of XML repo file – based on the one available from freedesktop.org's shared MIME package.

          Okay, so if someone gets a chance please run a small crawl with this in the next few days and let us know how it works. Otherwise, I'll do the same myself in a couple days and if there are no objections, I'd like to commit this then.

          Show
          Chris A. Mattmann added a comment - - edited Initial patch for comments: 1. This patch removes the MimeType system, and its associated java src files, config files and unit tests from Nutch. This information is in Tika now and is replaced by its TIka counterparts. 2. This patch uses the unreleased 0.1-dev version of Tika. When 0.1 is officially released, we can convert to that, though I don't anticipate any MimeType API changes between now and then. 3. All unit tests for core and plugins pass, however, it's probably a good idea to run at least a small crawl with this patch and see if everything works fine. I don't really have the time for this now, so anyone want to try? (cough cough Dogacan cough cough ) 4. It's worth noting that this MimeType system from Tika changes the traditional Nutch mime type system (IMO for the better) in a couple of ways. First, whereas the old MimeType system was very happy to return null in places where it couldn't figure out the MimeType, this system tries to return a "default" MimeType (which in this case is "application/octet-stream") if it can't guess the mime type from those that it knows about. Second, this mime type system uses a different type of XML repo file – based on the one available from freedesktop.org's shared MIME package. Okay, so if someone gets a chance please run a small crawl with this in the next few days and let us know how it works. Otherwise, I'll do the same myself in a couple days and if there are no objections, I'd like to commit this then.
          Hide
          Chris A. Mattmann added a comment -

          Tika 0.1 unrelased jar file – drop this in $NUTCH_SRC_HOME/lib

          Show
          Chris A. Mattmann added a comment - Tika 0.1 unrelased jar file – drop this in $NUTCH_SRC_HOME/lib
          Hide
          Chris A. Mattmann added a comment -
          • Applied patch, with minor changes to use static version of MimeUtils Tika interface, and to only instantiate once per object family
          • Tested on small crawl of apache.org sites, mime type set appropriately
          Show
          Chris A. Mattmann added a comment - Applied patch, with minor changes to use static version of MimeUtils Tika interface, and to only instantiate once per object family Tested on small crawl of apache.org sites, mime type set appropriately
          Hide
          Chris A. Mattmann added a comment -
          • Patch applied to trunk in r583016
          Show
          Chris A. Mattmann added a comment - Patch applied to trunk in r583016
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Nutch-Nightly #231 (See http://lucene.zones.apache.org:8080/hudson/job/Nutch-Nightly/231/ )

            People

            • Assignee:
              Chris A. Mattmann
              Reporter:
              Chris A. Mattmann
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development