Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2216

speculation should normalize progress rates based on amount of input data

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • jobtracker
    • None

    Description

      We frequently see skews in data distribution both on the mappers and reducers. The small ones finish quickly and the longer ones immediately get speculated. We should normalize progress rates used by speculation with some metric correlated to the amount of data processed by the task (like bytes read of rows processed). That will prevent these unnecessary speculations.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-718 Support for per-phase speculative execution

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              jsensarma Joydeep Sen Sarma
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: