Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7183

Make app master recover history from latest history file that exists

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • applicationmaster
    • None

    Description

      When running a mapreduce job, when the original app master is killed, the new app master normally attempts to recover by reading the jhist file that was written by the app master from the previous app attempt (e.g. current app attempt - 1).

      This is usually fine, but is a problem in the following situation:

      1. App master 1 writes history to jobid_1.jhist, then is killed
      2. App master 2 starts up but is killed before it has the chance to write any history to jobid_2.jhist
      3. App master 3 attempts to recover, but it can't find jobid_2.jhist, so all job progress is lost.

      This problem manifests as "Unable to parse prior job history, aborting recovery" and "Could not parse the old history file. Will not have old AMinfos" errors, all job progress being lost, and previous app attempts not showing up in the job history UI.

      To fix this problem, if jobid_2.jhist is missing, app master 3 should just recover using the history in jobid_1.jhist.

      Related JIRAs that mention this same problem:

      https://issues.apache.org/jira/browse/MAPREDUCE-4729

      https://issues.apache.org/jira/browse/MAPREDUCE-4767 

      Attachments

        1. MAPREDUCE-7183.patch
          6 kB
          Mikayla Konst

        Activity

          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 16m 42s trunk passed
          +1 compile 0m 29s trunk passed
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 32s trunk passed
          +1 shadedclient 11m 36s branch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 45s trunk passed
          +1 javadoc 0m 23s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 26s the patch passed
          +1 compile 0m 21s the patch passed
          +1 javac 0m 21s the patch passed
          -0 checkstyle 0m 13s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: The patch generated 3 new + 46 unchanged - 1 fixed = 49 total (was 47)
          +1 mvnsite 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 12m 5s patch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 51s the patch passed
          +1 javadoc 0m 21s the patch passed
                Other Tests
          -1 unit 1m 0s hadoop-mapreduce-client-common in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          47m 35s



          Reason Tests
          Failed junit tests hadoop.mapreduce.v2.jobhistory.TestJobHistoryUtils



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f
          JIRA Issue MAPREDUCE-7183
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12957838/MAPREDUCE-7183.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 5e8da2366fc9 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6aa6345
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_191
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7580/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7580/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7580/testReport/
          Max. process+thread count 441 (vs. ulimit of 10000)
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7580/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 42s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 32s trunk passed +1 shadedclient 11m 36s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 45s trunk passed +1 javadoc 0m 23s trunk passed       Patch Compile Tests +1 mvninstall 0m 26s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed -0 checkstyle 0m 13s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: The patch generated 3 new + 46 unchanged - 1 fixed = 49 total (was 47) +1 mvnsite 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 5s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 21s the patch passed       Other Tests -1 unit 1m 0s hadoop-mapreduce-client-common in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 47m 35s Reason Tests Failed junit tests hadoop.mapreduce.v2.jobhistory.TestJobHistoryUtils Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f JIRA Issue MAPREDUCE-7183 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12957838/MAPREDUCE-7183.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 5e8da2366fc9 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 6aa6345 maven version: Apache Maven 3.3.9 Default Java 1.8.0_191 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7580/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7580/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7580/testReport/ Max. process+thread count 441 (vs. ulimit of 10000) modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7580/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 48s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 28m 26s trunk passed
          +1 compile 0m 35s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 shadedclient 13m 54s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 2s trunk passed
          +1 javadoc 0m 30s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 31s the patch passed
          +1 javac 0m 31s the patch passed
          -0 checkstyle 0m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: The patch generated 3 new + 46 unchanged - 1 fixed = 49 total (was 47)
          +1 mvnsite 0m 34s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 14m 54s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 6s the patch passed
          +1 javadoc 0m 23s the patch passed
                Other Tests
          -1 unit 1m 5s hadoop-mapreduce-client-common in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          66m 25s



          Reason Tests
          Failed junit tests hadoop.mapreduce.v2.jobhistory.TestJobHistoryUtils



          Subsystem Report/Notes
          Docker Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4
          JIRA Issue MAPREDUCE-7183
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12957838/MAPREDUCE-7183.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 51a6eddf7184 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / e872ceb
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_222
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7643/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7643/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7643/testReport/
          Max. process+thread count 307 (vs. ulimit of 5500)
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7643/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 48s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 28m 26s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 36s trunk passed +1 shadedclient 13m 54s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 2s trunk passed +1 javadoc 0m 30s trunk passed       Patch Compile Tests +1 mvninstall 0m 30s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -0 checkstyle 0m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: The patch generated 3 new + 46 unchanged - 1 fixed = 49 total (was 47) +1 mvnsite 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 14m 54s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 6s the patch passed +1 javadoc 0m 23s the patch passed       Other Tests -1 unit 1m 5s hadoop-mapreduce-client-common in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 66m 25s Reason Tests Failed junit tests hadoop.mapreduce.v2.jobhistory.TestJobHistoryUtils Subsystem Report/Notes Docker Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 JIRA Issue MAPREDUCE-7183 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12957838/MAPREDUCE-7183.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 51a6eddf7184 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / e872ceb maven version: Apache Maven 3.3.9 Default Java 1.8.0_222 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7643/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7643/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7643/testReport/ Max. process+thread count 307 (vs. ulimit of 5500) modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7643/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.

          People

            mkonst Mikayla Konst
            mkonst Mikayla Konst
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: