Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7278

Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0, 3.4.0
    • 3.3.0, 3.2.2, 3.4.0
    • task
    • None

    Description

      When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER state for some time, we observe two task attempts are launched simultaneously even when speculative execution is disabled.

      This results in the below message shown in the killed attempts, indicating speculation has occurred. This is an issue for jobs which require speculative execution to be strictly disabled.

       

       

       

      Attachments

        1. MAPREDUCE-7278.004.patch
          9 kB
          Tarun Parimi
        2. MAPREDUCE-7278.003.patch
          8 kB
          Tarun Parimi
        3. MAPREDUCE-7278.002.patch
          3 kB
          Tarun Parimi
        4. MAPREDUCE-7278.001.patch
          3 kB
          Tarun Parimi
        5. Screen Shot 2020-04-30 at 8.04.27 PM.png
          40 kB
          Tarun Parimi

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6485 MR job hanged forever because all resources are taken up by reducers and the last map attempt never get resource to run

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              tarunparimi Tarun Parimi
              tarunparimi Tarun Parimi
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: