Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-961

Expose Tika's boilerpipe support

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.11
    • 1.12
    • parser
    • None
    • Patch Available

    Description

      Tika 0.8 comes with the Boilerpipe content handler which can be used to extract boilerplate content from HTML pages. We should see how we can expose Boilerplate in the Nutch cofiguration.

      Use the following properties to enable and control Boilerpipe.

      <property>
  <name>tika.extractor</name>
  <value>none</value>
  <description>
  Which text extraction algorithm to use. Valid values are: boilerpipe or none.
  </description>
</property>
 
<property> 
  <name>tika.extractor.boilerpipe.algorithm</name>
  <value>ArticleExtractor</value>
  <description> 
  Which Boilerpipe algorithm to use. Valid values are: DefaultExtractor, ArticleExtractor
  or CanolaExtractor.
  </description>
</property>

      Attachments

        1. NUTCH-961v2.patch
          3 kB
          Gabriele Kahlout
        2. NUTCH-961-2.1-v2.patch
          7 kB
          Roland von Herget
        3. NUTCH-961-2.1-v1.patch
          7 kB
          Roland von Herget
        4. NUTCH-961-1.8-1.patch
          7 kB
          Markus Jelsma
        5. NUTCH-961-1.5-1.patch
          7 kB
          Markus Jelsma
        6. NUTCH-961-1.4-dombuilder-1.patch
          0.6 kB
          Markus Jelsma
        7. NUTCH-961-1.3-tikaparser1.patch
          3 kB
          Gabriele Kahlout
        8. NUTCH-961-1.3-tikaparser.patch
          2 kB
          Markus Jelsma
        9. NUTCH-961-1.3-3.patch
          2 kB
          Markus Jelsma
        10. NUTCH-961-1.11-1.patch
          7 kB
          Vincent Slot
        11. NUTCH-961.patch
          3 kB
          Markus Jelsma
        12. NUTCH-961.patch
          6 kB
          Markus Jelsma
        13. nutch-2.x-boilerpipe.patch
          5 kB
          Alexander Kingson
        14. BoilerpipeExtractorRepository.java
          3 kB
          Markus Jelsma

        Issue Links

          depends upon

          Bug - A problem which impairs or prevents the functions of the product. TIKA-676 Boilerpipe fails

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. NUTCH-967 Upgrade to Tika 0.9

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. NUTCH-1375 extract main content of a html file

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. NUTCH-1233 Rely on Tika for outlink extraction

          • Major - Major loss of function.
          • Closed

          Activity

            People

              markus17 Markus Jelsma
              markus17 Markus Jelsma
              Votes:
              6 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: