Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6579

JobStatus#getFailureInfo should not output diagnostic information when the job is running

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      From https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt TestNetworkedJob are failed intermittently.

      Running org.apache.hadoop.mapred.TestNetworkedJob
      Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
      testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time elapsed: 30.55 sec  <<< FAILURE!
      org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 +0000 2015] Application is Activated, waiting for resources to be assigned for AM.  Details : AM Partition = <DEFAULT_PARTITION> ; Partition Resource = <memory:8192, vCores:16> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> but was:<[]>
      	at org.junit.Assert.assertEquals(Assert.java:115)
      	at org.junit.Assert.assertEquals(Assert.java:144)
      	at org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
      
      1. MAPREDUCE-6579.01.patch
        3 kB
        Akira Ajisaka
      2. MAPREDUCE-6579.02.patch
        11 kB
        Akira Ajisaka
      3. MAPREDUCE-6579.03.patch
        11 kB
        Akira Ajisaka
      4. MAPREDUCE-6579.04.patch
        11 kB
        Akira Ajisaka
      5. MAPREDUCE-6579.05.patch
        11 kB
        Akira Ajisaka
      6. MAPREDUCE-6579.06.patch
        3 kB
        Akira Ajisaka
      7. MAPREDUCE-6579.07.patch
        11 kB
        Naganarasimha G R
      8. MAPREDUCE-6579.08.patch
        11 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          ajisakaa Akira Ajisaka added a comment -

          I found YARN-3946 is the cause by running git bisect. I'll look into the issue.

          Show
          ajisakaa Akira Ajisaka added a comment - I found YARN-3946 is the cause by running git bisect. I'll look into the issue.
          Hide
          ajisakaa Akira Ajisaka added a comment - - edited

          After YARN-3946, JobClient.getFailureInfo() returns the application master's diagnostic message. Therefore the following code fails:

                assertEquals( runningJob.getFailureInfo(),"");
          

          I'm thinking the code can be replaced by

              String diag = runningJob.getFailureInfo();
              assertTrue(diag.startsWith(AMState.ACTIVATED) || diag.startsWith(AMState.ASSIGNED));
          

          because the code intend to verify that the job is not failing.

          By the way, the following javadoc is no longer correct. I'll update this.

          o.a.h.mapreduce.JobStatus
             /**
              * Gets any available info on the reason of failure of the job.
              * @return diagnostic information on why a job might have failed.
              */
             public synchronized String getFailureInfo() {
               return this.failureInfo;
             }
          

          Furthermore, I'd like to add a new method getDiagnostics to JobStatus and deprecate getFailureInfo because the method name getFailureInfo is confusing. I'll create a separate jira for this.

          Show
          ajisakaa Akira Ajisaka added a comment - - edited After YARN-3946 , JobClient.getFailureInfo() returns the application master's diagnostic message. Therefore the following code fails: assertEquals( runningJob.getFailureInfo(),""); I'm thinking the code can be replaced by String diag = runningJob.getFailureInfo(); assertTrue(diag.startsWith(AMState.ACTIVATED) || diag.startsWith(AMState.ASSIGNED)); because the code intend to verify that the job is not failing. By the way, the following javadoc is no longer correct. I'll update this. o.a.h.mapreduce.JobStatus /** * Gets any available info on the reason of failure of the job. * @ return diagnostic information on why a job might have failed. */ public synchronized String getFailureInfo() { return this .failureInfo; } Furthermore, I'd like to add a new method getDiagnostics to JobStatus and deprecate getFailureInfo because the method name getFailureInfo is confusing. I'll create a separate jira for this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch to fix the test failure.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch to fix the test failure.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 18s trunk passed
          +1 compile 3m 15s trunk passed with JDK v1.8.0_66
          +1 compile 3m 21s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 36s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 42s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 50s the patch passed
          +1 compile 3m 5s the patch passed with JDK v1.8.0_66
          +1 javac 3m 5s the patch passed
          +1 compile 3m 18s the patch passed with JDK v1.7.0_91
          +1 javac 3m 18s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 1s Patch has no whitespace issues.
          +1 findbugs 1m 56s the patch passed
          +1 javadoc 0m 32s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 38s the patch passed with JDK v1.7.0_91
          +1 unit 2m 2s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          -1 unit 118m 8s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          +1 unit 2m 29s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          -1 unit 120m 56s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 27s Patch generated 15 ASF License warnings.
          277m 26s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl
          JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780505/MAPREDUCE-6579.01.patch
          JIRA Issue MAPREDUCE-6579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cfaf476abd5f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 96d8f1d
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 18s trunk passed +1 compile 3m 15s trunk passed with JDK v1.8.0_66 +1 compile 3m 21s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 36s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 42s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 50s the patch passed +1 compile 3m 5s the patch passed with JDK v1.8.0_66 +1 javac 3m 5s the patch passed +1 compile 3m 18s the patch passed with JDK v1.7.0_91 +1 javac 3m 18s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 1s Patch has no whitespace issues. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 32s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 38s the patch passed with JDK v1.7.0_91 +1 unit 2m 2s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. -1 unit 118m 8s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. +1 unit 2m 29s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. -1 unit 120m 56s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 27s Patch generated 15 ASF License warnings. 277m 26s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780505/MAPREDUCE-6579.01.patch JIRA Issue MAPREDUCE-6579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cfaf476abd5f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 96d8f1d Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6253/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          -1 findbugs

          The warnings are unrelated to the patch. I attached a patch to fix them in MAPREDUCE-6595.

          TestMapReduceJobControl timed out

          This is unrelated to the patch because the patch modifies only the test code.

          -1 asflicense

          This is also unrelated to the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - -1 findbugs The warnings are unrelated to the patch. I attached a patch to fix them in MAPREDUCE-6595 . TestMapReduceJobControl timed out This is unrelated to the patch because the patch modifies only the test code. -1 asflicense This is also unrelated to the patch.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Akira Ajisaka,
          Overall the fix is fine but only one issue (not related to your modifications), arguments for the assertEquals are swapped (first is expected and seconds arg is actual). If you can accomodate in this patch then fine even if not its ok !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Akira Ajisaka , Overall the fix is fine but only one issue (not related to your modifications), arguments for the assertEquals are swapped (first is expected and seconds arg is actual). If you can accomodate in this patch then fine even if not its ok !
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Naganarasimha G R for the comment.

          arguments for the assertEquals are swapped

          I've noticed that but kept as-is because this issue is to fix the test failure.

          If you can accomodate in this patch then fine even if not its ok !

          Okay, I'll update the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Naganarasimha G R for the comment. arguments for the assertEquals are swapped I've noticed that but kept as-is because this issue is to fix the test failure. If you can accomodate in this patch then fine even if not its ok ! Okay, I'll update the patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          02 patch

          • Corrected the order of the arguments for assertEquals method.
          • Fixed the following useless assertions.
                assertEquals(info, info);
                assertEquals(info.toString(), info.toString());
            
          • Implemented equals method in BlackListInfo class to pass the following assertion.
                assertEquals(info, info2);
            
          Show
          ajisakaa Akira Ajisaka added a comment - 02 patch Corrected the order of the arguments for assertEquals method. Fixed the following useless assertions. assertEquals(info, info); assertEquals(info.toString(), info.toString()); Implemented equals method in BlackListInfo class to pass the following assertion. assertEquals(info, info2);
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 44s trunk passed
          +1 compile 2m 56s trunk passed with JDK v1.8.0_66
          +1 compile 3m 5s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 35s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 41s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 52s the patch passed
          +1 compile 3m 12s the patch passed with JDK v1.8.0_66
          +1 javac 3m 12s the patch passed
          +1 compile 3m 30s the patch passed with JDK v1.7.0_91
          +1 javac 3m 30s the patch passed
          -1 checkstyle 0m 22s Patch generated 1 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client (total was 91, now 92).
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 24s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core introduced 1 new FindBugs issues.
          -1 javadoc 1m 27s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was 100, now 100).
          +1 javadoc 0m 34s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 43s the patch passed with JDK v1.7.0_91
          +1 unit 2m 14s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          -1 unit 103m 37s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          +1 unit 2m 21s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 unit 105m 15s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_91.
          -1 asflicense 0m 28s Patch generated 15 ASF License warnings.
          246m 38s



          Reason Tests
          FindBugs module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
            org.apache.hadoop.mapred.ClusterStatus$BlackListInfo defines equals and uses Object.hashCode() At ClusterStatus.java:Object.hashCode() At ClusterStatus.java:[lines 177-185]
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780742/MAPREDUCE-6579.02.patch
          JIRA Issue MAPREDUCE-6579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 54262dea9ab4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2d16f40
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/new-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.html
          javadoc hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 44s trunk passed +1 compile 2m 56s trunk passed with JDK v1.8.0_66 +1 compile 3m 5s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 35s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 41s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 52s the patch passed +1 compile 3m 12s the patch passed with JDK v1.8.0_66 +1 javac 3m 12s the patch passed +1 compile 3m 30s the patch passed with JDK v1.7.0_91 +1 javac 3m 30s the patch passed -1 checkstyle 0m 22s Patch generated 1 new checkstyle issues in hadoop-mapreduce-project/hadoop-mapreduce-client (total was 91, now 92). +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 24s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core introduced 1 new FindBugs issues. -1 javadoc 1m 27s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was 100, now 100). +1 javadoc 0m 34s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 43s the patch passed with JDK v1.7.0_91 +1 unit 2m 14s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. -1 unit 103m 37s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. +1 unit 2m 21s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 unit 105m 15s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_91. -1 asflicense 0m 28s Patch generated 15 ASF License warnings. 246m 38s Reason Tests FindBugs module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core   org.apache.hadoop.mapred.ClusterStatus$BlackListInfo defines equals and uses Object.hashCode() At ClusterStatus.java:Object.hashCode() At ClusterStatus.java: [lines 177-185] JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780742/MAPREDUCE-6579.02.patch JIRA Issue MAPREDUCE-6579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 54262dea9ab4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2d16f40 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/new-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.html javadoc hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6255/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Akira Ajisaka, Thanks for considering and correcting the parameters other issues mentioned. Just that the check style and findbugs are same and are technical correction though it might not have any impact apart from it other corrections are fine !

          Show
          Naganarasimha Naganarasimha G R added a comment - Akira Ajisaka , Thanks for considering and correcting the parameters other issues mentioned. Just that the check style and findbugs are same and are technical correction though it might not have any impact apart from it other corrections are fine !
          Hide
          ajisakaa Akira Ajisaka added a comment -

          03 patch:

          • Added hashCode method in BlackListInfo class.
          • Added null check to equals method in BlackListInfo class.
          Show
          ajisakaa Akira Ajisaka added a comment - 03 patch: Added hashCode method in BlackListInfo class. Added null check to equals method in BlackListInfo class.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          +1 Patch LGTM !

          Show
          Naganarasimha Naganarasimha G R added a comment - +1 Patch LGTM !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 1s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 18s trunk passed
          +1 compile 3m 13s trunk passed with JDK v1.8.0_66
          +1 compile 3m 20s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          -1 findbugs 1m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 37s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 41s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 50s the patch passed
          +1 compile 3m 7s the patch passed with JDK v1.8.0_66
          +1 javac 3m 7s the patch passed
          +1 compile 3m 18s the patch passed with JDK v1.7.0_91
          +1 javac 3m 18s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 55s the patch passed
          -1 javadoc 1m 22s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was 100, now 100).
          +1 javadoc 0m 32s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 38s the patch passed with JDK v1.7.0_91
          +1 unit 2m 3s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          -1 unit 107m 16s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          +1 unit 2m 28s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
          +1 unit 104m 32s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_91.
          -1 asflicense 0m 28s Patch generated 15 ASF License warnings.
          250m 12s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapred.TestLocalMRNotification
            hadoop.mapreduce.v2.TestMRJobsWithProfiler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780898/MAPREDUCE-6579.03.patch
          JIRA Issue MAPREDUCE-6579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7e19fbc6f995 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 791c163
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          javadoc hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 18s trunk passed +1 compile 3m 13s trunk passed with JDK v1.8.0_66 +1 compile 3m 20s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 30s trunk passed -1 findbugs 1m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 37s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 41s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 50s the patch passed +1 compile 3m 7s the patch passed with JDK v1.8.0_66 +1 javac 3m 7s the patch passed +1 compile 3m 18s the patch passed with JDK v1.7.0_91 +1 javac 3m 18s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 55s the patch passed -1 javadoc 1m 22s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was 100, now 100). +1 javadoc 0m 32s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 38s the patch passed with JDK v1.7.0_91 +1 unit 2m 3s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. -1 unit 107m 16s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. +1 unit 2m 28s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 unit 104m 32s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_91. -1 asflicense 0m 28s Patch generated 15 ASF License warnings. 250m 12s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapred.TestLocalMRNotification   hadoop.mapreduce.v2.TestMRJobsWithProfiler Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780898/MAPREDUCE-6579.03.patch JIRA Issue MAPREDUCE-6579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7e19fbc6f995 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 791c163 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html javadoc hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6256/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          seems like tests and findbugs are unrelated to the modifications?

          Show
          Naganarasimha Naganarasimha G R added a comment - seems like tests and findbugs are unrelated to the modifications?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          seems like tests and findbugs are unrelated to the modifications?

          Yes. Thank you for your review!

          Show
          ajisakaa Akira Ajisaka added a comment - seems like tests and findbugs are unrelated to the modifications? Yes. Thank you for your review!
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Akira Ajisaka/Naganarasimha G R,

          A couple of comments:

          • Do you think we should rename JobStatus#getFailureInfo? Diagnostic message doesn't always means failure, JobStatus's InterfaceStability marked to "Evolving", we can update it.
          • I would suggest not compare the diagnostic info, it is evolving, only human is supposed to read this message and we have test in YARN already.
            175	      String diag = runningJob.getFailureInfo();
            176	      assertTrue("The state of the ApplicationMaster should be activated," +
            177	          "assigned, or launched.",
            178	          diag.contains(AMState.ACTIVATED.getDiagnosticMessage()) ||
            179	          diag.contains(AMState.ASSIGNED.getDiagnosticMessage()) ||
            180	          diag.contains(AMState.LAUNCHED.getDiagnosticMessage()));
            

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Akira Ajisaka / Naganarasimha G R , A couple of comments: Do you think we should rename JobStatus#getFailureInfo? Diagnostic message doesn't always means failure, JobStatus's InterfaceStability marked to "Evolving", we can update it. I would suggest not compare the diagnostic info, it is evolving, only human is supposed to read this message and we have test in YARN already. 175 String diag = runningJob.getFailureInfo(); 176 assertTrue( "The state of the ApplicationMaster should be activated," + 177 "assigned, or launched." , 178 diag.contains(AMState.ACTIVATED.getDiagnosticMessage()) || 179 diag.contains(AMState.ASSIGNED.getDiagnosticMessage()) || 180 diag.contains(AMState.LAUNCHED.getDiagnosticMessage())); Thoughts?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Do you think we should rename JobStatus#getFailureInfo? Diagnostic message doesn't always means failure, JobStatus's InterfaceStability marked to "Evolving", we can update it.

          I don't think we should rename JobStatus#getFailureInfo because Apache Tez is overriding this method.
          https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/DAGJobStatus.java

            @Override
            public synchronized String getFailureInfo() {
              return Joiner.on(". ").join(dagStatus.getDiagnostics());
            }
          

          Insteadly, we can deprecate the method in trunk/branch-2 and remove it in trunk. I'm thinking it should be done in a separate jira.

          I would suggest not compare the diagnostic info, it is evolving, only human is supposed to read this message and we have test in YARN already.

          Agree. I'll remove the assertion.

          Show
          ajisakaa Akira Ajisaka added a comment - Do you think we should rename JobStatus#getFailureInfo? Diagnostic message doesn't always means failure, JobStatus's InterfaceStability marked to "Evolving", we can update it. I don't think we should rename JobStatus#getFailureInfo because Apache Tez is overriding this method. https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/DAGJobStatus.java @Override public synchronized String getFailureInfo() { return Joiner.on( ". " ).join(dagStatus.getDiagnostics()); } Insteadly, we can deprecate the method in trunk/branch-2 and remove it in trunk. I'm thinking it should be done in a separate jira. I would suggest not compare the diagnostic info, it is evolving, only human is supposed to read this message and we have test in YARN already. Agree. I'll remove the assertion.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Tan, Wangda for the comments,
          bq, Do you think we should rename JobStatus#getFailureInfo?
          Yes & No, yes because we are setting diaginfo now even if there is no failure so it makes sense to update the name but no because this method was introduced though 4.5 yrs ago as part of MAPREDUCE-2952 still the interface is kept as evolving, so as Akira Ajisaka mentioned many other applications like tez might be using it. Even i would support this modification to be done in trunk.

          I would suggest not compare the diagnostic info, it is evolving, only human is supposed to read this message and we have test in YARN already.

          I am ok even if its not there, here were just comparing there is no error state information so it might be prone to have test case failure issues in future if new states are added .

          Thanks Akira Ajisaka for the new Patch, it LGTM !

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Tan, Wangda for the comments, bq, Do you think we should rename JobStatus#getFailureInfo? Yes & No, yes because we are setting diaginfo now even if there is no failure so it makes sense to update the name but no because this method was introduced though 4.5 yrs ago as part of MAPREDUCE-2952 still the interface is kept as evolving, so as Akira Ajisaka mentioned many other applications like tez might be using it. Even i would support this modification to be done in trunk. I would suggest not compare the diagnostic info, it is evolving, only human is supposed to read this message and we have test in YARN already. I am ok even if its not there, here were just comparing there is no error state information so it might be prone to have test case failure issues in future if new states are added . Thanks Akira Ajisaka for the new Patch, it LGTM !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 37s Maven dependency ordering for branch
          +1 mvninstall 8m 10s trunk passed
          +1 compile 2m 3s trunk passed with JDK v1.8.0_66
          +1 compile 2m 8s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          -1 findbugs 1m 19s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
          +1 javadoc 0m 46s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 47s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 1m 50s the patch passed with JDK v1.8.0_66
          +1 javac 1m 50s the patch passed
          +1 compile 1m 57s the patch passed with JDK v1.7.0_91
          +1 javac 1m 57s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 13s the patch passed
          +1 javadoc 0m 36s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 39s the patch passed with JDK v1.7.0_91
          +1 unit 2m 9s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
          -1 unit 101m 24s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          -1 unit 2m 16s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_91.
          +1 unit 105m 19s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_91.
          -1 asflicense 0m 27s Patch generated 15 ASF License warnings.
          243m 8s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.7.0_91 Failed junit tests hadoop.mapreduce.tools.TestCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784906/MAPREDUCE-6579.04.patch
          JIRA Issue MAPREDUCE-6579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 99df4c28a24e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 662e17b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 37s Maven dependency ordering for branch +1 mvninstall 8m 10s trunk passed +1 compile 2m 3s trunk passed with JDK v1.8.0_66 +1 compile 2m 8s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 31s trunk passed -1 findbugs 1m 19s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 46s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 47s trunk passed with JDK v1.7.0_91 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 1m 50s the patch passed with JDK v1.8.0_66 +1 javac 1m 50s the patch passed +1 compile 1m 57s the patch passed with JDK v1.7.0_91 +1 javac 1m 57s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 13s the patch passed +1 javadoc 0m 36s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 39s the patch passed with JDK v1.7.0_91 +1 unit 2m 9s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. -1 unit 101m 24s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. -1 unit 2m 16s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_91. +1 unit 105m 19s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_91. -1 asflicense 0m 27s Patch generated 15 ASF License warnings. 243m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.7.0_91 Failed junit tests hadoop.mapreduce.tools.TestCLI Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784906/MAPREDUCE-6579.04.patch JIRA Issue MAPREDUCE-6579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 99df4c28a24e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 662e17b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6292/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Agree, latest patch LGTM, thanks!

          Show
          leftnoteasy Wangda Tan added a comment - Agree, latest patch LGTM, thanks!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Naganarasimha G R and Wangda Tan for the reviews.
          Hi Wangda, could you give binding +1?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Naganarasimha G R and Wangda Tan for the reviews. Hi Wangda, could you give binding +1?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Seems like this failure is shown frequently in other jira's, Akira Ajisaka can you get it committed as Tan, Wangda has agreed as +1, else Rohith Sharma K S can you get this committed ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Seems like this failure is shown frequently in other jira's, Akira Ajisaka can you get it committed as Tan, Wangda has agreed as +1, else Rohith Sharma K S can you get this committed ?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Looking into this JIRA, had offline sync up with Naganarasimha G R.
          Earlier to YARN-3946, JobStatus#getFailureInfo used to return empty string if job is running. And used to set valid diagnosis message when the job is failed. But now, it is change in the behavior that API returns diagnosis message if job is running. API return type and name are not in sync and change in return type.

          I think we should keep MR API return as old behavior only. Any clients like Tez using this API would get affected. Any thoughts?

          And as a side note, if fix is not in test case then JIRA summary can be updated as per real issue.

          Show
          rohithsharma Rohith Sharma K S added a comment - Looking into this JIRA, had offline sync up with Naganarasimha G R . Earlier to YARN-3946 , JobStatus#getFailureInfo used to return empty string if job is running. And used to set valid diagnosis message when the job is failed. But now, it is change in the behavior that API returns diagnosis message if job is running. API return type and name are not in sync and change in return type. I think we should keep MR API return as old behavior only. Any clients like Tez using this API would get affected. Any thoughts? And as a side note, if fix is not in test case then JIRA summary can be updated as per real issue.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Rohith Sharma K S for sharing your thoughts, One approach i can think of is checking the status of job in the MR client side and only if its failed we can fetch and return the diagnosis message from yarn else we can return empty string. This will ensure the compatability is not broken if any client application is checking the failure message with out checking the job status.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Rohith Sharma K S for sharing your thoughts, One approach i can think of is checking the status of job in the MR client side and only if its failed we can fetch and return the diagnosis message from yarn else we can return empty string. This will ensure the compatability is not broken if any client application is checking the failure message with out checking the job status.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          And as a side note, if fix is not in test case then JIRA summary can be updated as per real issue.

          I meat if agreed upon my thoughts then it is required to change the summary.

          Show
          rohithsharma Rohith Sharma K S added a comment - And as a side note, if fix is not in test case then JIRA summary can be updated as per real issue. I meat if agreed upon my thoughts then it is required to change the summary.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          One approach i can think of is checking the status of job in the MR client side and only if its failed we can fetch and return the diagnosis message from yarn else we can return empty string. This will ensure the compatability is not broken if any client application is checking the failure message with out checking the job status.

          Sounds good. I'll update the patch. I'll update and the summary of this issue as well.

          Show
          ajisakaa Akira Ajisaka added a comment - One approach i can think of is checking the status of job in the MR client side and only if its failed we can fetch and return the diagnosis message from yarn else we can return empty string. This will ensure the compatability is not broken if any client application is checking the failure message with out checking the job status. Sounds good. I'll update the patch. I'll update and the summary of this issue as well.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Raised the priority to blocker since the issue is to keep API compatibility.

          Show
          ajisakaa Akira Ajisaka added a comment - Raised the priority to blocker since the issue is to keep API compatibility.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 33s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 1m 24s trunk passed with JDK v1.8.0_72
          +1 compile 1m 38s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 1m 32s trunk passed
          +1 javadoc 0m 32s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 39s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 47s the patch passed
          +1 compile 1m 21s the patch passed with JDK v1.8.0_72
          +1 javac 1m 21s the patch passed
          +1 compile 1m 36s the patch passed with JDK v1.7.0_95
          +1 javac 1m 36s the patch passed
          -1 checkstyle 0m 23s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 4 new + 89 unchanged - 1 fixed = 93 total (was 90)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 30s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 35s the patch passed with JDK v1.7.0_95
          +1 unit 1m 56s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72.
          -1 unit 103m 53s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_72.
          +1 unit 2m 20s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
          +1 unit 104m 32s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 31s Patch generated 15 ASF License warnings.
          237m 58s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787092/MAPREDUCE-6579.05.patch
          JIRA Issue MAPREDUCE-6579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e2aeb6820af9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fba6e9f
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 1m 24s trunk passed with JDK v1.8.0_72 +1 compile 1m 38s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 32s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 39s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 47s the patch passed +1 compile 1m 21s the patch passed with JDK v1.8.0_72 +1 javac 1m 21s the patch passed +1 compile 1m 36s the patch passed with JDK v1.7.0_95 +1 javac 1m 36s the patch passed -1 checkstyle 0m 23s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 4 new + 89 unchanged - 1 fixed = 93 total (was 90) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 30s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 35s the patch passed with JDK v1.7.0_95 +1 unit 1m 56s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72. -1 unit 103m 53s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_72. +1 unit 2m 20s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 unit 104m 32s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 31s Patch generated 15 ASF License warnings. 237m 58s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787092/MAPREDUCE-6579.05.patch JIRA Issue MAPREDUCE-6579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e2aeb6820af9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fba6e9f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6316/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          The patch mostly reasonable to me. Only doubt I have is I am not pretty sure how was the getFailureInfo was behaving for success/killed/failed. Current patch returns empty string for failed/killed. Should it return empty string for success state?
          Jason Lowe would you please provide your thoughts/opinion this.

          Show
          rohithsharma Rohith Sharma K S added a comment - The patch mostly reasonable to me. Only doubt I have is I am not pretty sure how was the getFailureInfo was behaving for success/killed/failed. Current patch returns empty string for failed/killed. Should it return empty string for success state? Jason Lowe would you please provide your thoughts/opinion this.
          Hide
          jlowe Jason Lowe added a comment -

          It's a little unfortunate that YARN-3946 started putting non-fatal messages into what is typically an app-driven diagnostic repository. Now all applications will start getting these (probably mostly annoying) messages for every job completion, assuming that most app frameworks dump the diagnostic strings when the application completes. It seems these new messages only make sense to report when the job is active and are mostly noise afterwards.

          Back to the MapReduce side of this, IMHO we need to return diagnostics for any case where we used to return diagnostics before. Since this is specific to MapReduce, we can check the MR AM to see all the places where we could set a diagnostic. Most places I found only set the diagnostic when the job fails, but I did find at least one place where the diagnostic could be set yet the job could succeed. When a task fails a job diagnostic is added, see JobImpl.TaskCompletedTransition#taskFailed. If the user configured the job to allow some tasks to fail yet the job can succeed then we could end up with a successful job with some task failure messages in the diagnostics.

          However that's a relatively rare config for a typical MapReduce job, and I'm not sure how many downstream software stacks are going to start getting upset when they see getFailureInfo start returning data on a regular basis for successful jobs. It's rather unfortunate that the method is called getFailureInfo and will now always contain messages unrelated to any failure. The downstream stacks should be checking the overall job status and not empty/non-empty on the getFailureInfo result to know whether the job really did fail or not, so on one hand I'm leaning towards reporting them on success as well. But then part of me thinks it will simply be annoying to have every job dump a bunch of messages on waiting to schedule, waiting to register, etc. on every successful job, which leads me to wonder if we really want YARN-3946 to work the way it does.

          Show
          jlowe Jason Lowe added a comment - It's a little unfortunate that YARN-3946 started putting non-fatal messages into what is typically an app-driven diagnostic repository. Now all applications will start getting these (probably mostly annoying) messages for every job completion, assuming that most app frameworks dump the diagnostic strings when the application completes. It seems these new messages only make sense to report when the job is active and are mostly noise afterwards. Back to the MapReduce side of this, IMHO we need to return diagnostics for any case where we used to return diagnostics before. Since this is specific to MapReduce, we can check the MR AM to see all the places where we could set a diagnostic. Most places I found only set the diagnostic when the job fails, but I did find at least one place where the diagnostic could be set yet the job could succeed. When a task fails a job diagnostic is added, see JobImpl.TaskCompletedTransition#taskFailed. If the user configured the job to allow some tasks to fail yet the job can succeed then we could end up with a successful job with some task failure messages in the diagnostics. However that's a relatively rare config for a typical MapReduce job, and I'm not sure how many downstream software stacks are going to start getting upset when they see getFailureInfo start returning data on a regular basis for successful jobs. It's rather unfortunate that the method is called getFailureInfo and will now always contain messages unrelated to any failure. The downstream stacks should be checking the overall job status and not empty/non-empty on the getFailureInfo result to know whether the job really did fail or not, so on one hand I'm leaning towards reporting them on success as well. But then part of me thinks it will simply be annoying to have every job dump a bunch of messages on waiting to schedule, waiting to register, etc. on every successful job, which leads me to wonder if we really want YARN-3946 to work the way it does.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks Jason for sharing your opinion.

          It seems these new messages only make sense to report when the job is active and are mostly noise afterwards.

          very true

          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks Jason for sharing your opinion. It seems these new messages only make sense to report when the job is active and are mostly noise afterwards. very true
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I think fix should be done at YARN so that it will not break any downstream projects.
          One of the approach at YARN I can think of is reset the RM added diagnosis messages to empty string once app is in final_saving state. Any thoughts?

          Show
          rohithsharma Rohith Sharma K S added a comment - I think fix should be done at YARN so that it will not break any downstream projects. One of the approach at YARN I can think of is reset the RM added diagnosis messages to empty string once app is in final_saving state. Any thoughts?
          Hide
          sunilg Sunil G added a comment -

          Yes Rohith Sharma K S. From YARN if diagnostics can be cleared out once app moves to final state, we could avoid such scenarios. Meantime we would have already collected enough reasons for apps stuck in ACCEPTED or SCHEDULED state. And in final states, I see there is no need to keep such informations as we already pass that checkpoint.

          Show
          sunilg Sunil G added a comment - Yes Rohith Sharma K S . From YARN if diagnostics can be cleared out once app moves to final state, we could avoid such scenarios. Meantime we would have already collected enough reasons for apps stuck in ACCEPTED or SCHEDULED state. And in final states, I see there is no need to keep such informations as we already pass that checkpoint.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Jason Lowe, Rohith Sharma K S, and Sunil G for your comments. Mostly agree with you. I have a question:

          From YARN if diagnostics can be cleared out once app moves to final state, we could avoid such scenarios.

          What is final state? The test asserts that the diagnostics is cleared out when the app is running.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Jason Lowe , Rohith Sharma K S , and Sunil G for your comments. Mostly agree with you. I have a question: From YARN if diagnostics can be cleared out once app moves to final state, we could avoid such scenarios. What is final state? The test asserts that the diagnostics is cleared out when the app is running.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          06 patch (sample) :

          • Clear diagnostic information when an AM is registered.
          • Wait until the AM is registered in TestNetworkedJob
          Show
          ajisakaa Akira Ajisaka added a comment - 06 patch (sample) : Clear diagnostic information when an AM is registered. Wait until the AM is registered in TestNetworkedJob
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 34s trunk passed
          +1 compile 5m 39s trunk passed with JDK v1.8.0_72
          +1 compile 6m 44s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          +1 findbugs 1m 34s trunk passed
          +1 javadoc 0m 34s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 41s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 7m 13s the patch passed with JDK v1.8.0_72
          +1 javac 7m 13s the patch passed
          +1 compile 7m 4s the patch passed with JDK v1.7.0_95
          +1 javac 7m 4s the patch passed
          +1 checkstyle 1m 4s the patch passed
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 35s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 44s the patch passed with JDK v1.7.0_95
          -1 unit 70m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
          -1 unit 100m 43s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_72.
          -1 unit 72m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 unit 104m 5s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 32s Patch generated 15 ASF License warnings.
          395m 52s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787640/MAPREDUCE-6579.06.patch
          JIRA Issue MAPREDUCE-6579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0cb463b748ff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2fb423e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 34s trunk passed +1 compile 5m 39s trunk passed with JDK v1.8.0_72 +1 compile 6m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 10s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 1m 34s trunk passed +1 javadoc 0m 34s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 41s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 7m 13s the patch passed with JDK v1.8.0_72 +1 javac 7m 13s the patch passed +1 compile 7m 4s the patch passed with JDK v1.7.0_95 +1 javac 7m 4s the patch passed +1 checkstyle 1m 4s the patch passed +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 35s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 44s the patch passed with JDK v1.7.0_95 -1 unit 70m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. -1 unit 100m 43s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_72. -1 unit 72m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 unit 104m 5s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 32s Patch generated 15 ASF License warnings. 395m 52s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787640/MAPREDUCE-6579.06.patch JIRA Issue MAPREDUCE-6579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0cb463b748ff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2fb423e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6319/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Jason Lowe,Sunil G & Akira Ajisaka, for sharing your views but we have already taken care of what you guys have suggested. YARN-3946 takes care of resetting the diagnostic message once the AM is registered already in RMAppAttemptImpl.AMRegisteredTransition

          @@ -1507,6 +1515,9 @@ public void transition(RMAppAttemptImpl appAttempt,
                 appAttempt.originalTrackingUrl =
                     sanitizeTrackingUrl(registrationEvent.getTrackingurl());
           
          +      // reset AMLaunchDiagnostics once AM Registers with RM
          +      appAttempt.updateAMLaunchDiagnostics(null);
          +
          

          Jason Lowe,
          As we have ensured that the Am launch Diagnostic info is reset once the AM is registered, do you see any further problems with the approach taken in YARN-3946 ? IMHO it was very useful information to get this statistics and it was apt to put it into App's Diagnostic information instead of creating a new AM Launch diagnostic and i have taken care such that if anywhere diagnostic is set then RMAppImpl or RMAppAttemptImpl will not return the AM Launch diagnostic but return the actual message.
          Further in Map reduce, currently its assumed that the diagnostics are always failure diagnostics which IMO is wrong. As per MAPREDUCE-6579.05.patch JobStatus is as follows

              public synchronized String getFailureInfo() {
          -     return this.failureInfo;
          +     if (runState == State.FAILED || runState == State.KILLED) {
          +       return this.failureInfo;
          +     }
          +     return "";
              }
          

          As you were informing this might not be correct place to do this check, whether would it be better to handle in NotRunningJob.getJobReport ?

          Akira Ajisaka,
          As per modifications in MAPREDUCE-6579.06, you have reset the diagnostics in RMAppAttemptImpl.AMRegisteredTransition just after the location i have reset. I commented that code and ran the test case, it ran successfully. So IMO the modification in the testcase is fine but RMAppAttemptImpl modifications are not required and at the same time we need to discuss whether we need to add additional check in NotRunningJob.getJobReport

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Jason Lowe , Sunil G & Akira Ajisaka , for sharing your views but we have already taken care of what you guys have suggested. YARN-3946 takes care of resetting the diagnostic message once the AM is registered already in RMAppAttemptImpl.AMRegisteredTransition @@ -1507,6 +1515,9 @@ public void transition(RMAppAttemptImpl appAttempt, appAttempt.originalTrackingUrl = sanitizeTrackingUrl(registrationEvent.getTrackingurl()); + // reset AMLaunchDiagnostics once AM Registers with RM + appAttempt.updateAMLaunchDiagnostics( null ); + Jason Lowe , As we have ensured that the Am launch Diagnostic info is reset once the AM is registered, do you see any further problems with the approach taken in YARN-3946 ? IMHO it was very useful information to get this statistics and it was apt to put it into App's Diagnostic information instead of creating a new AM Launch diagnostic and i have taken care such that if anywhere diagnostic is set then RMAppImpl or RMAppAttemptImpl will not return the AM Launch diagnostic but return the actual message. Further in Map reduce, currently its assumed that the diagnostics are always failure diagnostics which IMO is wrong. As per MAPREDUCE-6579 .05.patch JobStatus is as follows public synchronized String getFailureInfo() { - return this .failureInfo; + if (runState == State.FAILED || runState == State.KILLED) { + return this .failureInfo; + } + return ""; } As you were informing this might not be correct place to do this check, whether would it be better to handle in NotRunningJob.getJobReport ? Akira Ajisaka , As per modifications in MAPREDUCE-6579 .06 , you have reset the diagnostics in RMAppAttemptImpl.AMRegisteredTransition just after the location i have reset. I commented that code and ran the test case, it ran successfully. So IMO the modification in the testcase is fine but RMAppAttemptImpl modifications are not required and at the same time we need to discuss whether we need to add additional check in NotRunningJob.getJobReport
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Naganarasimha G R for the detailed comment.

          IMO the modification in the testcase is fine but RMAppAttemptImpl modifications are not required

          Agree.

          whether we need to add additional check in NotRunningJob.getJobReport

          I'm okay to add the additional check for compatibility, but it is not enough. I'm thinking there are some possibility to add diagnostic message (such as task failure) even if the job is running. In this case, the following check is needed.

              public synchronized String getFailureInfo() {
          -     return this.failureInfo;
          +     if (runState == State.FAILED || runState == State.KILLED) {
          +       return this.failureInfo;
          +     }
          +     return "";
              }
          
          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Naganarasimha G R for the detailed comment. IMO the modification in the testcase is fine but RMAppAttemptImpl modifications are not required Agree. whether we need to add additional check in NotRunningJob.getJobReport I'm okay to add the additional check for compatibility, but it is not enough. I'm thinking there are some possibility to add diagnostic message (such as task failure) even if the job is running. In this case, the following check is needed. public synchronized String getFailureInfo() { - return this .failureInfo; + if (runState == State.FAILED || runState == State.KILLED) { + return this .failureInfo; + } + return ""; }
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Akira Ajisaka,
          Well IMO i thought the focus of this jira is only to ensure that the impact for YARN-3946 is nullified, i.e. if the YARN is sending some diagnostic message which is not related to failure, IMO that should not be considered as Job's failureInfo.
          But checked the documentation of JobStatus.getFailureInfo()

          /**
          Gets any available info on the reason of failure of the job

          • @return diagnostic information on why a job might have failed.
            */
            public synchronized String getFailureInfo()

          Hence i agree that it would better be handled as per the code in MAPREDUCE-6579.05.patch, so looks like no modifications are required if we have to set the failureInfo only when the job has failed. But just concerned about compatability.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Akira Ajisaka , Well IMO i thought the focus of this jira is only to ensure that the impact for YARN-3946 is nullified, i.e. if the YARN is sending some diagnostic message which is not related to failure, IMO that should not be considered as Job's failureInfo . But checked the documentation of JobStatus.getFailureInfo() /** Gets any available info on the reason of failure of the job @return diagnostic information on why a job might have failed. */ public synchronized String getFailureInfo() Hence i agree that it would better be handled as per the code in MAPREDUCE-6579 .05.patch, so looks like no modifications are required if we have to set the failureInfo only when the job has failed. But just concerned about compatability.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Apologize for the delays, I was on vacation in recent a couple of weeks.

          Went through discussions,

          For concerns from Jason Lowe:

          It's a little unfortunate that YARN-3946 started putting non-fatal messages into what is typically an app-driven diagnostic repository....

          It seems these new messages only make sense to report when the job is active and are mostly noise afterwards.

          As mentioned by Naganarasimha G R, diagnostic message added by YARN-3946 only exists before AM registers to RM, and it will be cleaned up after AM registers. So I think this will not pollute the final failure message from app.

          The only issue I can see is MR assumes diagnostic message == failure message: To make changes of YARN-3946 backward compatible to MR apps, I think 05-patch is the simplest way to fix the problem: it avoid taking any field modified by YARN-3946 and keeps everything else same.

          Thoughts? NGarla_Unused/Akira Ajisaka/Sunil G.

          Show
          leftnoteasy Wangda Tan added a comment - Apologize for the delays, I was on vacation in recent a couple of weeks. Went through discussions, For concerns from Jason Lowe : It's a little unfortunate that YARN-3946 started putting non-fatal messages into what is typically an app-driven diagnostic repository.... It seems these new messages only make sense to report when the job is active and are mostly noise afterwards. As mentioned by Naganarasimha G R , diagnostic message added by YARN-3946 only exists before AM registers to RM, and it will be cleaned up after AM registers. So I think this will not pollute the final failure message from app. The only issue I can see is MR assumes diagnostic message == failure message: To make changes of YARN-3946 backward compatible to MR apps, I think 05-patch is the simplest way to fix the problem: it avoid taking any field modified by YARN-3946 and keeps everything else same. Thoughts? NGarla_Unused / Akira Ajisaka / Sunil G .
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          hi Tan, Wangda
          If we want to just limit the scope of this jira to remove the impact of YARN-3946, then approach in 05.patch is not good enough as JobStatus is not only got from YARN when not active but can be got from the actual job(AM) too.
          And as Jason Lowe was mentioning there can be some scenarios where the job is trying to set the failure info though the job is successful, so do we want to address that in this jira ?
          I have uploaded a 07.patch which takes care of handling only the impact of YARN-3946, please take a look whether that is fine.

          Show
          Naganarasimha Naganarasimha G R added a comment - hi Tan, Wangda If we want to just limit the scope of this jira to remove the impact of YARN-3946 , then approach in 05.patch is not good enough as JobStatus is not only got from YARN when not active but can be got from the actual job(AM) too. And as Jason Lowe was mentioning there can be some scenarios where the job is trying to set the failure info though the job is successful, so do we want to address that in this jira ? I have uploaded a 07.patch which takes care of handling only the impact of YARN-3946 , please take a look whether that is fine.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 28s Maven dependency ordering for branch
          +1 mvninstall 9m 28s trunk passed
          +1 compile 2m 42s trunk passed with JDK v1.8.0_72
          +1 compile 2m 24s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 20s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 55s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 54s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 5s the patch passed
          +1 compile 2m 37s the patch passed with JDK v1.8.0_72
          +1 javac 2m 37s the patch passed
          +1 compile 2m 25s the patch passed with JDK v1.7.0_95
          +1 javac 2m 25s the patch passed
          +1 checkstyle 0m 30s the patch passed
          +1 mvnsite 1m 14s the patch passed
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 30s the patch passed
          +1 javadoc 0m 51s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 49s the patch passed with JDK v1.7.0_95
          -1 unit 3m 3s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72.
          -1 unit 146m 58s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_72.
          -1 unit 3m 6s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95.
          -1 unit 145m 32s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
          -1 asflicense 0m 33s Patch generated 15 ASF License warnings.
          335m 5s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat
            hadoop.mapred.TestFileInputFormat
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl
          JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat
            hadoop.mapred.TestFileInputFormat
            hadoop.mapred.TestLocalMRNotification
            hadoop.mapred.TestClusterMapReduceTestCase
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
            org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788234/MAPREDUCE-6579.07.patch
          JIRA Issue MAPREDUCE-6579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 247a22b2f9e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 453e7e0
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 28s Maven dependency ordering for branch +1 mvninstall 9m 28s trunk passed +1 compile 2m 42s trunk passed with JDK v1.8.0_72 +1 compile 2m 24s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 55s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 54s trunk passed with JDK v1.7.0_95 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 5s the patch passed +1 compile 2m 37s the patch passed with JDK v1.8.0_72 +1 javac 2m 37s the patch passed +1 compile 2m 25s the patch passed with JDK v1.7.0_95 +1 javac 2m 25s the patch passed +1 checkstyle 0m 30s the patch passed +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 30s the patch passed +1 javadoc 0m 51s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 49s the patch passed with JDK v1.7.0_95 -1 unit 3m 3s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72. -1 unit 146m 58s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_72. -1 unit 3m 6s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95. -1 unit 145m 32s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 33s Patch generated 15 ASF License warnings. 335m 5s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat   hadoop.mapred.TestFileInputFormat   hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat   hadoop.mapred.TestFileInputFormat   hadoop.mapred.TestLocalMRNotification   hadoop.mapred.TestClusterMapReduceTestCase JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption   org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788234/MAPREDUCE-6579.07.patch JIRA Issue MAPREDUCE-6579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 247a22b2f9e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 453e7e0 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6327/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Naganarasimha G R,

          I think we need to fix both:

          • JobStatus.getFailureInfo
          • NotRunningJob.getDiagnostics

          For original MR, diagnostic message can be only set when application finishes.

          Diagnostic message returned only job is failed or killed In the 07 patch, I think we should at least include finished state. I'm not sure if existing MR sets diagnostic message when job finishes successfully, but it is very possible that people will do it in the future.

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Hi Naganarasimha G R , I think we need to fix both: JobStatus.getFailureInfo NotRunningJob.getDiagnostics For original MR, diagnostic message can be only set when application finishes. Diagnostic message returned only job is failed or killed In the 07 patch, I think we should at least include finished state. I'm not sure if existing MR sets diagnostic message when job finishes successfully, but it is very possible that people will do it in the future. Thoughts?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Tan, Wangda for the reply,
          Modifications in JobStatus.getFailureInfo will ensure that neither AM(/Job) nor RM can return diagnostic info when the job is successful, but modifications i suggested in NotRunningJob.getDiagnostics tries to take care only of avoiding the RM diagnostics when app is not launched.
          As i said earlier its ideal to go for JobStatus.getFailureInfo modifications but not sure about the compatibility i.e. AM/Job is purposefully setting the diagnostic message even when the job is running or job has succeeded.
          Also IMO either one of the modifications is sufficient based on the scope what we are going to decide.
          thoughts?

          Diagnostic message returned only job is failed or killed In the 07 patch, I think we should at least include finished state.

          As per the description of YarnApplicationState.FINISHED is Application which finished successfully. If its successful and the diagnostic message set from YARN then thought we should avoid returning it, else we need again handle this part in JobStatus.getFailureInfo.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Tan, Wangda for the reply, Modifications in JobStatus.getFailureInfo will ensure that neither AM(/Job) nor RM can return diagnostic info when the job is successful, but modifications i suggested in NotRunningJob.getDiagnostics tries to take care only of avoiding the RM diagnostics when app is not launched. As i said earlier its ideal to go for JobStatus.getFailureInfo modifications but not sure about the compatibility i.e. AM/Job is purposefully setting the diagnostic message even when the job is running or job has succeeded. Also IMO either one of the modifications is sufficient based on the scope what we are going to decide. thoughts? Diagnostic message returned only job is failed or killed In the 07 patch, I think we should at least include finished state. As per the description of YarnApplicationState.FINISHED is Application which finished successfully . If its successful and the diagnostic message set from YARN then thought we should avoid returning it, else we need again handle this part in JobStatus.getFailureInfo .
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Oops i think i have assigned this to myself by mistake reassigning to Akira Ajisaka !

          Show
          Naganarasimha Naganarasimha G R added a comment - Oops i think i have assigned this to myself by mistake reassigning to Akira Ajisaka !
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Naganarasimha G R and Wangda Tan for the comments and updating the patch.

          • If we want to just limit the scope of this jira to remove the impact of YARN-3946, 07 patch looks good to me.
          • If we want to expand the scope to keep compatibility with original MR, we need to fix JobStatus.getFailureInfo as well.

          I'm thinking the latter should be discussed/fixed in a separate jira.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Naganarasimha G R and Wangda Tan for the comments and updating the patch. If we want to just limit the scope of this jira to remove the impact of YARN-3946 , 07 patch looks good to me. If we want to expand the scope to keep compatibility with original MR, we need to fix JobStatus.getFailureInfo as well. I'm thinking the latter should be discussed/fixed in a separate jira.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          I'm thinking the latter should be discussed/fixed in a separate jira.

          I am ok with this approach may be more discussions can happen in the new jira and we can close this jira with just removing the impact of YARN-3946

          Show
          Naganarasimha Naganarasimha G R added a comment - I'm thinking the latter should be discussed/fixed in a separate jira. I am ok with this approach may be more discussions can happen in the new jira and we can close this jira with just removing the impact of YARN-3946
          Hide
          sunilg Sunil G added a comment -

          Hi NGarla_Unused
          is it possible to get FAILED or KILLED in NotRunningJob.getDiagnostics?

          Show
          sunilg Sunil G added a comment - Hi NGarla_Unused is it possible to get FAILED or KILLED in NotRunningJob.getDiagnostics ?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R, Akira Ajisaka.

          I agree to only handle the impact of YARN-3946 in this JIRA.

          However, I still think we should show diagnostic message for completed applications. Without YARN-3946, diagnostic message is added to NotRunningJob in any case. We'd better not change that behavior.

          Even though, according to definition of YARN, FINISHED state means successfully. However, for individual application, it is possible that adding some information or explanation to the FINISHED state.

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , Akira Ajisaka . I agree to only handle the impact of YARN-3946 in this JIRA. However, I still think we should show diagnostic message for completed applications. Without YARN-3946 , diagnostic message is added to NotRunningJob in any case. We'd better not change that behavior. Even though, according to definition of YARN, FINISHED state means successfully. However, for individual application, it is possible that adding some information or explanation to the FINISHED state.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          I have not tried but based on the assumption that Before AM gets launched we can KILL the submitted application and may be due to containerlocalization or launch failures can make the Application status to FAILED.

          Show
          Naganarasimha Naganarasimha G R added a comment - I have not tried but based on the assumption that Before AM gets launched we can KILL the submitted application and may be due to containerlocalization or launch failures can make the Application status to FAILED .
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Tan, Wangda for the feedback,

          Even though, according to definition of YARN, FINISHED state means successfully. However, for individual application, it is possible that adding some information or explanation to the FINISHED state.

          Agree, Akira Ajisaka, can you please take care of this comment ? if required I can upload with the correction.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Tan, Wangda for the feedback, Even though, according to definition of YARN, FINISHED state means successfully. However, for individual application, it is possible that adding some information or explanation to the FINISHED state. Agree, Akira Ajisaka , can you please take care of this comment ? if required I can upload with the correction.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Naganarasimha G R and Wangda Tan for the feedback. Updated the patch to show diagnostics if the app is finished successfully.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Naganarasimha G R and Wangda Tan for the feedback. Updated the patch to show diagnostics if the app is finished successfully.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          +1 , LGTM for the latest patch !

          Show
          Naganarasimha Naganarasimha G R added a comment - +1 , LGTM for the latest patch !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 7m 27s trunk passed
          +1 compile 1m 30s trunk passed with JDK v1.8.0_72
          +1 compile 1m 46s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 37s trunk passed
          +1 javadoc 0m 34s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 42s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 1m 40s the patch passed with JDK v1.8.0_72
          +1 javac 1m 40s the patch passed
          +1 compile 1m 49s the patch passed with JDK v1.7.0_95
          +1 javac 1m 49s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 2s the patch passed
          +1 javadoc 0m 32s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 37s the patch passed with JDK v1.7.0_95
          -1 unit 2m 4s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72.
          -1 unit 107m 21s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_72.
          -1 unit 2m 17s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95.
          +1 unit 101m 47s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 36s Patch generated 19 ASF License warnings.
          240m 44s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat
            hadoop.mapred.TestFileInputFormat
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat
            hadoop.mapred.TestFileInputFormat



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788608/MAPREDUCE-6579.08.patch
          JIRA Issue MAPREDUCE-6579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0a76fe55465d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 748b6c0
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 7m 27s trunk passed +1 compile 1m 30s trunk passed with JDK v1.8.0_72 +1 compile 1m 46s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 0m 34s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 42s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 1m 40s the patch passed with JDK v1.8.0_72 +1 javac 1m 40s the patch passed +1 compile 1m 49s the patch passed with JDK v1.7.0_95 +1 javac 1m 49s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 0m 32s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 37s the patch passed with JDK v1.7.0_95 -1 unit 2m 4s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72. -1 unit 107m 21s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_72. -1 unit 2m 17s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95. +1 unit 101m 47s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 36s Patch generated 19 ASF License warnings. 240m 44s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat   hadoop.mapred.TestFileInputFormat   hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.lib.input.TestFileInputFormat   hadoop.mapred.TestFileInputFormat Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788608/MAPREDUCE-6579.08.patch JIRA Issue MAPREDUCE-6579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0a76fe55465d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 748b6c0 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6331/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Wangda Tan, would you review the latest patch?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Wangda Tan , would you review the latest patch?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Ping: Wangda Tan

          Show
          ajisakaa Akira Ajisaka added a comment - Ping: Wangda Tan
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Naganarasimha G R for reviewing the patch!
          Hi Wangda Tan, I need binding +1 to commit the latest patch, so I'm asking you to review it because you reviewed/committed YARN-3946, which broke this test.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Naganarasimha G R for reviewing the patch! Hi Wangda Tan , I need binding +1 to commit the latest patch, so I'm asking you to review it because you reviewed/committed YARN-3946 , which broke this test.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Akira Ajisaka,

          Sorry for my delays. I thought I have +1-ed already.

          Thanks for updates, +1 to latest patch. Will commit this shortly.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Akira Ajisaka , Sorry for my delays. I thought I have +1-ed already. Thanks for updates, +1 to latest patch. Will commit this shortly.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to trunk/branch-2/branch-2.8, thanks Akira Ajisaka and reviews from Naganarasimha G R/Rohith Sharma K S.

          Show
          leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2/branch-2.8, thanks Akira Ajisaka and reviews from Naganarasimha G R / Rohith Sharma K S .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9467 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9467/)
          MAPREDUCE-6579. JobStatus#getFailureInfo should not output diagnostic (wangda: rev 6529c87551e07e9da1e1b273eff4c6bad6b3e838)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ClusterStatus.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestNetworkedJob.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9467 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9467/ ) MAPREDUCE-6579 . JobStatus#getFailureInfo should not output diagnostic (wangda: rev 6529c87551e07e9da1e1b273eff4c6bad6b3e838) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ClusterStatus.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestNetworkedJob.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks a lot!

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks a lot!

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              rohithsharma Rohith Sharma K S
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development