Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.3.0
    • None
    • None
    • Reviewed

    Description

      org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing can fail because of race condition.

      testHistoryParsingWithParseErrors(org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing)  Time elapsed: 4.102 sec  <<< ERROR!
      java.io.IOException: Unable to initialize History Viewer
              at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
              at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:137)
              at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
              at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:798)
              at org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.<init>(JobHistoryParser.java:86)
              at org.apache.hadoop.mapreduce.jobhistory.HistoryViewer.<init>(HistoryViewer.java:85)
              at org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing.checkHistoryParsing(TestJobHistoryParsing.java:339)
              at org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing.testHistoryParsingWithParseErrors(TestJobHistoryParsing.java:125)
      

      In the checkHistoryParsing() function, after

      HistoryFileInfo fileInfo = jobHistory.getJobFileInfo(jobId);
      

      a thread named MoveIntermediateToDone will be launched to move history file from done_intermediate to done directory.
      If the history file is moved,

            HistoryViewer viewer = new HistoryViewer(fc.makeQualified(
                fileInfo.getHistoryFile()).toString(), conf, true);
      

      will throw IOException,because the history file is not found.

      Attachments

        1. MAPREDUCE-5679-3.diff
          5 kB
          Liyin Liang
        2. MAPREDUCE-5679-2.diff
          3 kB
          Liyin Liang
        3. MAPREDUCE-5679.diff
          0.9 kB
          Liyin Liang

        Activity

          liangly Liyin Liang added a comment -

          This patch encapsulates "test output for HistoryViewer" with lock of fileInfo to avoid race condition.

          liangly Liyin Liang added a comment - This patch encapsulates "test output for HistoryViewer" with lock of fileInfo to avoid race condition.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12618359/MAPREDUCE-5679.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4259//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4259//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618359/MAPREDUCE-5679.diff against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4259//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4259//console This message is automatically generated.
          liangly Liyin Liang added a comment -

          testHistoryParsingForFailedAttempts() and testCountersForFailedTask() have the same race conditions.

          liangly Liyin Liang added a comment - testHistoryParsingForFailedAttempts() and testCountersForFailedTask() have the same race conditions.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12618375/MAPREDUCE-5679-2.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4260//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4260//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618375/MAPREDUCE-5679-2.diff against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4260//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4260//console This message is automatically generated.

          Thanks for the patch, liangly. Patch looks good overall. Would like to see the code properly indented within the synchronized blocks to match the Hadoop coding guidelines.

          Another minor nit is that the code is holding the locks longer than necessary (only need to lock fileinfo until data is parsed) but not a must-fix since it's just test code.

          jlowe Jason Darrell Lowe added a comment - Thanks for the patch, liangly . Patch looks good overall. Would like to see the code properly indented within the synchronized blocks to match the Hadoop coding guidelines. Another minor nit is that the code is holding the locks longer than necessary (only need to lock fileinfo until data is parsed) but not a must-fix since it's just test code.
          liangly Liyin Liang added a comment -

          Incorporate Jason Lowe 's comment. Thanks for the review.

          liangly Liyin Liang added a comment - Incorporate Jason Lowe 's comment. Thanks for the review.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12618527/MAPREDUCE-5679-3.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4262//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4262//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618527/MAPREDUCE-5679-3.diff against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4262//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4262//console This message is automatically generated.

          +1 to the latest patch. Committing this.

          jlowe Jason Darrell Lowe added a comment - +1 to the latest patch. Committing this.

          Thanks, Liyin! I committed this to trunk and branch-2.

          jlowe Jason Darrell Lowe added a comment - Thanks, Liyin! I committed this to trunk and branch-2.
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4898 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4898/)
          MAPREDUCE-5679. TestJobHistoryParsing has race condition. Contributed by Liyin Liang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551616)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4898 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4898/ ) MAPREDUCE-5679 . TestJobHistoryParsing has race condition. Contributed by Liyin Liang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551616 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #425 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/425/)
          MAPREDUCE-5679. TestJobHistoryParsing has race condition. Contributed by Liyin Liang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551616)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #425 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/425/ ) MAPREDUCE-5679 . TestJobHistoryParsing has race condition. Contributed by Liyin Liang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551616 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1616 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1616/)
          MAPREDUCE-5679. TestJobHistoryParsing has race condition. Contributed by Liyin Liang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551616)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1616 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1616/ ) MAPREDUCE-5679 . TestJobHistoryParsing has race condition. Contributed by Liyin Liang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551616 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1642 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1642/)
          MAPREDUCE-5679. TestJobHistoryParsing has race condition. Contributed by Liyin Liang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551616)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1642 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1642/ ) MAPREDUCE-5679 . TestJobHistoryParsing has race condition. Contributed by Liyin Liang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551616 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java

          People

            liangly Liyin Liang
            liangly Liyin Liang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: