Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1309

fetch queue management

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 1.4
    • Fix Version/s: None
    • Component/s: fetcher
    • Labels:

      Description

      when run fetch in hadoop with multiple concurrent mapper, there are multiple independent fetchQueues that make hard to manage them. i suggest that construct fetchQueues before begin of run with this line:
      feeder = new QueueFeeder(input, fetchQueues, threadCount * 50);

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              behnam.nikbakht behnam nikbakht

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment