Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1347

fetcher politeness related to map-reduce

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.4
    • Fix Version/s: None
    • Component/s: fetcher
    • Labels:

      Description

      when Nutch is running on Hadoop , based on map-reduce concept, each map task do some thing on it's owned data, so, each fetcher map-task work with it's Queues and do not know any thing about other Queus. so, enforce delay between successive requests and maximum concurrent requests policies on it's Queues. but with a simple test we found that it's not good piliteness mechanism when we have multiple map tasks.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              behnam.nikbakht behnam nikbakht
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: