Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2637

Number of fetcher reducers is misconfigured when the arg not passed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3, 2.3.1
    • Fix Version/s: 2.4
    • Component/s: fetcher
    • Labels:
      None

      Description

      I'm kind a new to this, so sorry if i'm wrong.
      The thing is the number of fetcher reducers are currently set to the value of "mapreduce.job.maps" when the arg not passed. It should be "mapreduce.job.reduces".

       

      https://github.com/apache/nutch/blob/2.x/src/java/org/apache/nutch/fetcher/FetcherJob.java#L216

      Line: 216, branch-2.X

      if (numTasks == null || numTasks < 1) {
      currentJob.setNumReduceTasks(currentJob.getConfiguration().getInt(
      "mapreduce.job.maps", currentJob.getNumReduceTasks()));
      } else {
      currentJob.setNumReduceTasks(numTasks);
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                sinsi404 Fumio Nakajima
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: