Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3497

JobControlCompiler should only do reducer estimation when the job has a reduce phase

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 0.12.0
    • None
    • None

    Description

      Currently, JobControlCompiler makes an estimation for the number of reducers required (by default based on input size into mappers) regardless of whether there is a reduce phase in the job. This is unnecessary, especially when running more complex custom reducer estimators.

      Change to only estimate reducers when necessary.

      Attachments

        1. reducer_estimation.patch
          2 kB
          Akihiro Matsukawa

        Activity

          People

            amatsukawa Akihiro Matsukawa
            amatsukawa Akihiro Matsukawa
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: