Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Lately TestCLI has been failing sometimes in precommit builds:

      Running org.apache.hadoop.mapreduce.tools.TestCLI
      Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
      testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI)  Time elapsed: 0.037 sec  <<< FAILURE!
      java.lang.AssertionError: null
      	at org.junit.Assert.fail(Assert.java:86)
      	at org.junit.Assert.assertTrue(Assert.java:41)
      	at org.junit.Assert.assertTrue(Assert.java:52)
      	at org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
      
      1. mapreduce6625.001.patch
        0.9 kB
        Haibo Chen
      2. mapreduce6625.002.patch
        3 kB
        Haibo Chen
      3. mapreduce6625.003.patch
        3 kB
        Haibo Chen

        Issue Links

          Activity

          Hide
          haibochen Haibo Chen added a comment -

          The time elapsed is measured in millseconds (Time.monotonicNow()). In the first loop execution, max retry is set to 0, and no Thread.sleep() is executed. The execution of CLI.getJob() can finish within a millisecond, which may cause start=end (because of the rounding of nanoseconds to milliseconds in Time.monotonicNow()).

          Show
          haibochen Haibo Chen added a comment - The time elapsed is measured in millseconds (Time.monotonicNow()). In the first loop execution, max retry is set to 0, and no Thread.sleep() is executed. The execution of CLI.getJob() can finish within a millisecond, which may cause start=end (because of the rounding of nanoseconds to milliseconds in Time.monotonicNow()).
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 12s trunk passed
          +1 compile 0m 23s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 47s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvnsite 0m 26s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 53s the patch passed
          +1 javadoc 0m 19s the patch passed
          +1 unit 1m 59s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 14s The patch does not generate ASF License warnings.
          14m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812959/mapreduce6625.001.patch
          JIRA Issue MAPREDUCE-6625
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 07d1a4015e08 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0b9edf6
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6582/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6582/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 12s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 47s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 1m 59s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 14m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812959/mapreduce6625.001.patch JIRA Issue MAPREDUCE-6625 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 07d1a4015e08 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0b9edf6 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6582/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6582/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Seems like this test could also be flaky in the other direction. The very next assert is checking to make sure we didn't take too long to get the job, where too long is just 100 milliseconds. If the unit tests run on a loaded machine or the JVM otherwise pauses briefly for some reason this test could fail that way as well.

          Show
          jlowe Jason Lowe added a comment - Seems like this test could also be flaky in the other direction. The very next assert is checking to make sure we didn't take too long to get the job, where too long is just 100 milliseconds. If the unit tests run on a loaded machine or the JVM otherwise pauses briefly for some reason this test could fail that way as well.
          Hide
          haibochen Haibo Chen added a comment -

          Thanks Jason for pointing out the other issue as well. We could just remove the second assert as it may not be meaningful. But it seems to me that checking if retry inside CLI.getJob() works by measuring time elapsed because of sleep is not generally a robust solution. Instead, we can just rely on the return value. Attaching a patch that checks return value to see if retry works.

          Show
          haibochen Haibo Chen added a comment - Thanks Jason for pointing out the other issue as well. We could just remove the second assert as it may not be meaningful. But it seems to me that checking if retry inside CLI.getJob() works by measuring time elapsed because of sleep is not generally a robust solution. Instead, we can just rely on the return value. Attaching a patch that checks return value to see if retry works.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 37s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 6s trunk passed
          +1 compile 0m 26s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 30s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 53s trunk passed
          +1 javadoc 0m 24s trunk passed
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 23s the patch passed
          +1 javac 0m 23s the patch passed
          -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23)
          +1 mvnsite 0m 26s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 57s the patch passed
          +1 javadoc 0m 20s the patch passed
          +1 unit 2m 5s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          27m 23s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813164/mapreduce6625.002.patch
          JIRA Issue MAPREDUCE-6625
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d48a7a9eb282 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bf74dbf
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6585/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6585/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6585/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 37s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 6s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 53s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 24s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 57s the patch passed +1 javadoc 0m 20s the patch passed +1 unit 2m 5s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 27m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813164/mapreduce6625.002.patch JIRA Issue MAPREDUCE-6625 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d48a7a9eb282 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf74dbf Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6585/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6585/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6585/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          haibochen Haibo Chen added a comment -

          fixed the checkstyle issue

          Show
          haibochen Haibo Chen added a comment - fixed the checkstyle issue
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 15m 35s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 52s trunk passed
          +1 compile 0m 23s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 49s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 14s the patch passed
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 52s the patch passed
          +1 javadoc 0m 19s the patch passed
          +1 unit 2m 1s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          31m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813171/mapreduce6625.003.patch
          JIRA Issue MAPREDUCE-6625
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7a46a54876f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bf74dbf
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6586/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6586/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 15m 35s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 52s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 52s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 2m 1s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 31m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813171/mapreduce6625.003.patch JIRA Issue MAPREDUCE-6625 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7a46a54876f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf74dbf Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6586/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6586/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          +1

          Show
          rkanter Robert Kanter added a comment - +1
          Hide
          rkanter Robert Kanter added a comment -

          Thanks Haibo Chen. Committed to trunk and branch-2!

          Show
          rkanter Robert Kanter added a comment - Thanks Haibo Chen . Committed to trunk and branch-2!
          Hide
          jlowe Jason Lowe added a comment -

          Thanks Robert Kanter for picking this up – it dropped off my radar after vacations and conferences.

          Is there a reason this wasn't committed to 2.8? It looks like the original race was added in 2.8 via MAPREDUCE-6566, so it seems appropriate to fix it there as well. Otherwise those running the unit tests on 2.8-based releases will occasionally see this issue.

          Show
          jlowe Jason Lowe added a comment - Thanks Robert Kanter for picking this up – it dropped off my radar after vacations and conferences. Is there a reason this wasn't committed to 2.8? It looks like the original race was added in 2.8 via MAPREDUCE-6566 , so it seems appropriate to fix it there as well. Otherwise those running the unit tests on 2.8-based releases will occasionally see this issue.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10082 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10082/)
          MAPREDUCE-6625. TestCLI#testGetJob fails occasionally (haibochen via (rkanter: rev ad256beb50e8851d5e6e5e743b00e819e75e3549)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10082 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10082/ ) MAPREDUCE-6625 . TestCLI#testGetJob fails occasionally (haibochen via (rkanter: rev ad256beb50e8851d5e6e5e743b00e819e75e3549) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
          Hide
          rkanter Robert Kanter added a comment -

          Is there a reason this wasn't committed to 2.8?

          Jason Lowe, no real reason. I've just committed it to branch-2.8 too.

          Show
          rkanter Robert Kanter added a comment - Is there a reason this wasn't committed to 2.8? Jason Lowe , no real reason. I've just committed it to branch-2.8 too.

            People

            • Assignee:
              haibochen Haibo Chen
              Reporter:
              jlowe Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development