Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1024

Dynamically set fetchInterval by MIME-type

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 1.6
    • generator
    • None
    • Patch Available

    Description

      Add facility to configure default or fixed fetchInterval values by MIME-type. This is useful for conserving resources for files that are known to change frequently or never and everything in between.

      • simple key\tvalue\n configuration file
      • only set fetchInterval for new documents
      • keep max fetchInterval fixed by current config

      Attachments

        1. AdaptiveFetchSchedule.patch
          0.5 kB
          Markus Jelsma
        2. Nutch.patch
          0.5 kB
          Markus Jelsma
        3. adaptive-mimetypes.txt
          0.1 kB
          Markus Jelsma
        4. MimeAdaptiveFetchSchedule.java
          8 kB
          Markus Jelsma
        5. NUTCH-1024-1.5-1.patch
          14 kB
          Markus Jelsma
        6. NUTCH-1024-1.5-2.patch
          15 kB
          Markus Jelsma
        7. NUTCH-1024-1.5-3.patch
          17 kB
          Markus Jelsma

        Issue Links

          depends upon

          New Feature - A new feature of the product, which has yet to be developed. NUTCH-779 Mechanism for passing metadata from parse to crawldb

          • Major - Major loss of function.
          • Closed

          Activity

            People

              markus17 Markus Jelsma
              markus17 Markus Jelsma
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: