Nutch
  1. Nutch
  2. NUTCH-1024

Dynamically set fetchInterval by MIME-type

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6
    • Component/s: generator
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      Add facility to configure default or fixed fetchInterval values by MIME-type. This is useful for conserving resources for files that are known to change frequently or never and everything in between.

      • simple key\tvalue\n configuration file
      • only set fetchInterval for new documents
      • keep max fetchInterval fixed by current config
      1. NUTCH-1024-1.5-3.patch
        17 kB
        Markus Jelsma
      2. NUTCH-1024-1.5-2.patch
        15 kB
        Markus Jelsma
      3. NUTCH-1024-1.5-1.patch
        14 kB
        Markus Jelsma
      4. MimeAdaptiveFetchSchedule.java
        8 kB
        Markus Jelsma
      5. adaptive-mimetypes.txt
        0.1 kB
        Markus Jelsma
      6. Nutch.patch
        0.5 kB
        Markus Jelsma
      7. AdaptiveFetchSchedule.patch
        0.5 kB
        Markus Jelsma

        Issue Links

          Activity

          Lewis John McGibbney made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue depends upon NUTCH-779 [ NUTCH-779 ]
          Gavin made changes -
          Link This issue depends on NUTCH-779 [ NUTCH-779 ]
          Markus Jelsma made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Markus Jelsma made changes -
          Fix Version/s 1.6 [ 12319941 ]
          Fix Version/s 1.5 [ 12318246 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1024-1.5-3.patch [ 12520591 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1024-1.5-3.patch [ 12520588 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1024-1.5-3.patch [ 12520588 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1024-1.5-2.patch [ 12520393 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1024-1.5-1.patch [ 12516805 ]
          Markus Jelsma made changes -
          Fix Version/s 1.5 [ 12318246 ]
          Fix Version/s 1.4 [ 12316519 ]
          Markus Jelsma made changes -
          Attachment adaptive-mimetypes.txt [ 12490963 ]
          Markus Jelsma made changes -
          Attachment MimeAdaptiveFetchSchedule.java [ 12491190 ]
          Markus Jelsma made changes -
          Attachment adaptive-mimetypes.txt [ 12491205 ]
          Attachment MimeAdaptiveFetchSchedule.java [ 12491206 ]
          Markus Jelsma made changes -
          Attachment MimeAdaptiveFetchSchedule.java [ 12490964 ]
          Markus Jelsma made changes -
          Attachment MimeAdaptiveFetchSchedule.java [ 12491190 ]
          Markus Jelsma made changes -
          Component/s generator [ 12311358 ]
          Fix Version/s 1.4 [ 12316519 ]
          Fix Version/s 2.0 [ 12314893 ]
          Patch Info [Patch Available]
          Markus Jelsma made changes -
          Attachment adaptive-mimetypes.txt [ 12490963 ]
          Attachment MimeAdaptiveFetchSchedule.java [ 12490964 ]
          Attachment AdaptiveFetchSchedule.patch [ 12490965 ]
          Attachment Nutch.patch [ 12490966 ]
          Markus Jelsma made changes -
          Assignee Markus Jelsma [ markus17 ]
          Markus Jelsma made changes -
          Field Original Value New Value
          Link This issue depends on NUTCH-779 [ NUTCH-779 ]
          Markus Jelsma created issue -

            People

            • Assignee:
              Markus Jelsma
              Reporter:
              Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development