Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1347

fetcher politeness related to map-reduce

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • 1.4
    • None
    • fetcher

    Description

      when Nutch is running on Hadoop , based on map-reduce concept, each map task do some thing on it's owned data, so, each fetcher map-task work with it's Queues and do not know any thing about other Queus. so, enforce delay between successive requests and maximum concurrent requests policies on it's Queues. but with a simple test we found that it's not good piliteness mechanism when we have multiple map tasks.

      Attachments

        Activity

          People

            Unassigned Unassigned
            behnam.nikbakht behnam nikbakht
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: