Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7109

On completion of shuffle phase in reducers, mappers should not be launched again

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Currently, in actOnUnusableNodes in JobImpl mappers are re launched irrespective of whether their output is further required or not. This leads to unnecessary compute. Idea is to keep track of all shuffle phases of reducer and not relaunch mapper if shuffle phase in reducers is already done

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5817 Mappers get rescheduled on node transition even after all reducers are completed

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              kartik.bhatia Kartik Bhatia
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: