Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5271

ATS client doesn't work with Jersey 2 on the classpath

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      A workaround to avoid dependency conflict with Spark2, before a full classpath isolation solution is implemented.
      Skip instantiating a Timeline Service client if encountering NoClassDefFoundError.
      Show
      A workaround to avoid dependency conflict with Spark2, before a full classpath isolation solution is implemented. Skip instantiating a Timeline Service client if encountering NoClassDefFoundError.

      Description

      see SPARK-15343 : once Jersey 2 is on the CP, you can't instantiate a timeline client, even if the server is an ATS1.5 server and publishing is via the FS

      1. YARN-5271.01.patch
        4 kB
        Weiwei Yang
      2. YARN-5271.02.patch
        4 kB
        Weiwei Yang
      3. YARN-5271.branch-2.01.patch
        4 kB
        Weiwei Yang
      4. YARN-5271-branch-2.8.01.patch
        4 kB
        Weiwei Yang

        Issue Links

          Activity

          Hide
          cheersyang Weiwei Yang added a comment -

          Hi Steve Loughran

          I am trying to help on this one.
          It looks like this Jersey client was only used for ATS v2, I already saw a flag timelineServiceV2 in TimelineClientImpl class, can we only init this instance when the flag is true? So in theory a timeline client would work with Jersey 2 but talk to ATS v1 using the file system writer?

          Show
          cheersyang Weiwei Yang added a comment - Hi Steve Loughran I am trying to help on this one. It looks like this Jersey client was only used for ATS v2, I already saw a flag timelineServiceV2 in TimelineClientImpl class, can we only init this instance when the flag is true? So in theory a timeline client would work with Jersey 2 but talk to ATS v1 using the file system writer?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ATS 1.5 Is the filesystem outputter; ATSv1.0 is REST only. I don't know about anything else ... not been keeping current with this

          Show
          stevel@apache.org Steve Loughran added a comment - ATS 1.5 Is the filesystem outputter; ATSv1.0 is REST only. I don't know about anything else ... not been keeping current with this
          Hide
          cheersyang Weiwei Yang added a comment -

          I am assigning this one to myself now to investigate.

          Show
          cheersyang Weiwei Yang added a comment - I am assigning this one to myself now to investigate.
          Hide
          cheersyang Weiwei Yang added a comment -

          Attached a patch to resolve this. The thought is when timeline client failed to init, it doesn't necessarily fail the entire Yarn client. Like this case, client uses a higher version of jersey which is not compatible with yarn runtime. When this happens, we should print a warning on console and disable the timeline client, but let rest of yarn client run. Instead of failing the application.

          Show
          cheersyang Weiwei Yang added a comment - Attached a patch to resolve this. The thought is when timeline client failed to init, it doesn't necessarily fail the entire Yarn client. Like this case, client uses a higher version of jersey which is not compatible with yarn runtime. When this happens, we should print a warning on console and disable the timeline client, but let rest of yarn client run. Instead of failing the application.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 9s trunk passed
          +1 compile 0m 20s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 31s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          +1 findbugs 0m 41s trunk passed
          +1 javadoc 0m 18s trunk passed
          +1 mvninstall 0m 26s the patch passed
          +1 compile 0m 23s the patch passed
          +1 javac 0m 23s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 29s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 49s the patch passed
          +1 javadoc 0m 16s the patch passed
          +1 unit 16m 19s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          32m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833410/YARN-5271.01.patch
          JIRA Issue YARN-5271
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b96d6404961c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8a9f663
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13398/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13398/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 9s trunk passed +1 compile 0m 20s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 0m 41s trunk passed +1 javadoc 0m 18s trunk passed +1 mvninstall 0m 26s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 49s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 16m 19s hadoop-yarn-client in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 32m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833410/YARN-5271.01.patch JIRA Issue YARN-5271 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b96d6404961c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8a9f663 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13398/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/13398/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jo_desmet@yahoo.com Jo Desmet added a comment - - edited

          Note that such tight coupling should not be required, not so that we have that library dependency between spark and Yarn. The idea of Jersey is exactly to avoid this type of tight coupling between two application frameworks. I think it is still a huge drawback of not having the History Server up. I think it is 1./ Spark's job to have enough decoupling in place; 2./ Yarn to evolve fast enough, being able to use latest libraries and benefit from added features and improvements. That being said, most likely this fix is probably a good first step.

          Show
          jo_desmet@yahoo.com Jo Desmet added a comment - - edited Note that such tight coupling should not be required, not so that we have that library dependency between spark and Yarn. The idea of Jersey is exactly to avoid this type of tight coupling between two application frameworks. I think it is still a huge drawback of not having the History Server up. I think it is 1./ Spark's job to have enough decoupling in place; 2./ Yarn to evolve fast enough, being able to use latest libraries and benefit from added features and improvements. That being said, most likely this fix is probably a good first step.
          Hide
          cheersyang Weiwei Yang added a comment -

          Can somebody help to review this?

          Show
          cheersyang Weiwei Yang added a comment - Can somebody help to review this?
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hi Weiwei Yang thanks for the patch.
          While I think the fix itself is good, the unit test part fails to show regression if I remove the fix.

          I suspect you want to init/start/stop spyClient instead of client object:

                spyClient.init(conf);
                spyClient.start();
                assertFalse("Timeline client should be disabled when"
                    + "it is failed to init",
                    spyClient.timelineServiceEnabled);
                spyClient.stop();
          

          Also some nitpicking:

          LOG.warn("Timeline client could not be initialized "
          	            + "because dependency missing or incompatible,"
          

          It reads more natural to me to say "because of missing or incompatible dependency,".

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hi Weiwei Yang thanks for the patch. While I think the fix itself is good, the unit test part fails to show regression if I remove the fix. I suspect you want to init/start/stop spyClient instead of client object: spyClient.init(conf); spyClient.start(); assertFalse( "Timeline client should be disabled when" + "it is failed to init" , spyClient.timelineServiceEnabled); spyClient.stop(); Also some nitpicking: LOG.warn( "Timeline client could not be initialized " + "because dependency missing or incompatible," It reads more natural to me to say "because of missing or incompatible dependency,".
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Hi Wei-Chiu Chuang

          Thanks for looking at this one. You are absolutely right I should use spyClient instead of the client, I have fixed that in v2 patch (one for trunk and one for branch-2.8). I tested if I remove the code fix, the UT will fail like following,

          Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.558 sec <<< FAILURE! - in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
          testTimelineClientInitFailure(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)  Time elapsed: 0.465 sec  <<< ERROR!
          java.lang.NoClassDefFoundError: Mock a failure when init timeline instance
          	at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testTimelineClientInitFailure(TestYarnClient.java:170)
          

          Also I have revised the warning message according to your comment. Thank you!

          Show
          cheersyang Weiwei Yang added a comment - - edited Hi Wei-Chiu Chuang Thanks for looking at this one. You are absolutely right I should use spyClient instead of the client, I have fixed that in v2 patch (one for trunk and one for branch-2.8). I tested if I remove the code fix, the UT will fail like following, Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.558 sec <<< FAILURE! - in org.apache.hadoop.yarn.client.api.impl.TestYarnClient testTimelineClientInitFailure(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 0.465 sec <<< ERROR! java.lang.NoClassDefFoundError: Mock a failure when init timeline instance at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testTimelineClientInitFailure(TestYarnClient.java:170) Also I have revised the warning message according to your comment. Thank you!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 45s trunk passed
          +1 compile 0m 20s trunk passed
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 25s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 15s trunk passed
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 0m 14s the patch passed
          +1 unit 16m 37s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          29m 36s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5271
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838939/YARN-5271.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 610e63b07a5a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 43aef30
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13914/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13914/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 20s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 25s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 16m 37s hadoop-yarn-client in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 29m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5271 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838939/YARN-5271.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 610e63b07a5a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 43aef30 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13914/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/13914/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 10s branch-2.8 passed
          +1 compile 0m 19s branch-2.8 passed with JDK v1.8.0_101
          +1 compile 0m 22s branch-2.8 passed with JDK v1.7.0_111
          +1 checkstyle 0m 15s branch-2.8 passed
          +1 mvnsite 0m 25s branch-2.8 passed
          +1 mvneclipse 0m 14s branch-2.8 passed
          +1 findbugs 0m 37s branch-2.8 passed
          +1 javadoc 0m 15s branch-2.8 passed with JDK v1.8.0_101
          +1 javadoc 0m 16s branch-2.8 passed with JDK v1.7.0_111
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_101
          +1 javac 0m 17s the patch passed
          +1 compile 0m 18s the patch passed with JDK v1.7.0_111
          +1 javac 0m 18s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 22s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 47s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_101
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_111
          -1 unit 66m 6s hadoop-yarn-client in the patch failed with JDK v1.7.0_111.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          147m 19s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.yarn.client.TestGetGroups
            hadoop.yarn.client.api.impl.TestAMRMProxy
          JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient
          JDK v1.7.0_111 Failed junit tests hadoop.yarn.client.TestGetGroups
            hadoop.yarn.client.api.impl.TestAMRMProxy
          JDK v1.7.0_111 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5af2af1
          JIRA Issue YARN-5271
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838942/YARN-5271-branch-2.8.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1403e6cca942 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.8 / 0feb9bb
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13916/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13916/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13916/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 10s branch-2.8 passed +1 compile 0m 19s branch-2.8 passed with JDK v1.8.0_101 +1 compile 0m 22s branch-2.8 passed with JDK v1.7.0_111 +1 checkstyle 0m 15s branch-2.8 passed +1 mvnsite 0m 25s branch-2.8 passed +1 mvneclipse 0m 14s branch-2.8 passed +1 findbugs 0m 37s branch-2.8 passed +1 javadoc 0m 15s branch-2.8 passed with JDK v1.8.0_101 +1 javadoc 0m 16s branch-2.8 passed with JDK v1.7.0_111 +1 mvninstall 0m 19s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_101 +1 javac 0m 17s the patch passed +1 compile 0m 18s the patch passed with JDK v1.7.0_111 +1 javac 0m 18s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 47s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_111 -1 unit 66m 6s hadoop-yarn-client in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 147m 19s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.yarn.client.TestGetGroups   hadoop.yarn.client.api.impl.TestAMRMProxy JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient JDK v1.7.0_111 Failed junit tests hadoop.yarn.client.TestGetGroups   hadoop.yarn.client.api.impl.TestAMRMProxy JDK v1.7.0_111 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient Subsystem Report/Notes Docker Image:yetus/hadoop:5af2af1 JIRA Issue YARN-5271 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838942/YARN-5271-branch-2.8.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1403e6cca942 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 0feb9bb Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/13916/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13916/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/13916/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          The 02 patch looks good to me. Are there other comments?

          Show
          jojochuang Wei-Chiu Chuang added a comment - The 02 patch looks good to me. Are there other comments?
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Wei-Chiu Chuang

          Is there anything else you want me to work on for this patch? Please let me know.

          Show
          cheersyang Weiwei Yang added a comment - Hello Wei-Chiu Chuang Is there anything else you want me to work on for this patch? Please let me know.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks for reminder. +1 and will commit soon.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks for reminder. +1 and will commit soon.
          Hide
          cheersyang Weiwei Yang added a comment -

          Thank you!

          Show
          cheersyang Weiwei Yang added a comment - Thank you!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed this to trunk, branch-2 and branch-2.8. Thanks Weiwei Yang for the patch, Steve Loughran for filing the jira and Jo Desmet for comments.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed this to trunk, branch-2 and branch-2.8. Thanks Weiwei Yang for the patch, Steve Loughran for filing the jira and Jo Desmet for comments.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10860 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10860/)
          YARN-5271. ATS client doesn't work with Jersey 2 on the classpath. (weichiu: rev 09520cb439f8b002e3f2f3d8f5080ffc34f4bd5c)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10860 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10860/ ) YARN-5271 . ATS client doesn't work with Jersey 2 on the classpath. (weichiu: rev 09520cb439f8b002e3f2f3d8f5080ffc34f4bd5c) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
          Hide
          cheersyang Weiwei Yang added a comment -

          Great, thanks Wei-Chiu Chuang

          Show
          cheersyang Weiwei Yang added a comment - Great, thanks Wei-Chiu Chuang
          Hide
          gtCarrera9 Li Lu added a comment -

          Quick note: are we catching an error here and disables timeline service based on this? Catching errors seems to be inadequate as per Java API doc:

          An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions.

          (https://docs.oracle.com/javase/7/docs/api/java/lang/Error.html)

          Reopen this JIRA for more investigation.

          Show
          gtCarrera9 Li Lu added a comment - Quick note: are we catching an error here and disables timeline service based on this? Catching errors seems to be inadequate as per Java API doc: An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions. ( https://docs.oracle.com/javase/7/docs/api/java/lang/Error.html ) Reopen this JIRA for more investigation.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Li Lu

          Thank you for watching on this. This is a handler for an abnormal condition that client uses a conflict jersey dependency against Yarn, so I thought disabling timeline service was better than fail. I'd love to hear if you or anyone else have any other good suggestions.

          Show
          cheersyang Weiwei Yang added a comment - Hello Li Lu Thank you for watching on this. This is a handler for an abnormal condition that client uses a conflict jersey dependency against Yarn, so I thought disabling timeline service was better than fail. I'd love to hear if you or anyone else have any other good suggestions.
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks for the work Weiwei Yang! This looks like a pretty unfortunate case for uses to use the YARN client. I noticed we're not creating timeline clients if timeline service is turned off in the config. One inquiry is, can we fail fast and let the user disable timeline service? Raising errors as early as possible may avoid much troubles in the future?

          Show
          gtCarrera9 Li Lu added a comment - Thanks for the work Weiwei Yang ! This looks like a pretty unfortunate case for uses to use the YARN client. I noticed we're not creating timeline clients if timeline service is turned off in the config. One inquiry is, can we fail fast and let the user disable timeline service? Raising errors as early as possible may avoid much troubles in the future?
          Hide
          cheersyang Weiwei Yang added a comment -

          Hi Li Lu

          The point is you'll need to try catch this exception so you can fail with a more informative error message, something like "dependency conflict, you have to disable timeline server", and people will need to add "yarn.timeline-service.enabled=false" in their client code and start all over again. The fix here was trying to alleviate this pain, it prints a warning on console and warns user timeline client could not be initialized because of dependency issue, more user friendly. How anyone else thinks?

          Show
          cheersyang Weiwei Yang added a comment - Hi Li Lu The point is you'll need to try catch this exception so you can fail with a more informative error message, something like "dependency conflict, you have to disable timeline server", and people will need to add "yarn.timeline-service.enabled=false" in their client code and start all over again. The fix here was trying to alleviate this pain, it prints a warning on console and warns user timeline client could not be initialized because of dependency issue, more user friendly. How anyone else thinks?
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Weiwei Yang.

          The fix here was trying to alleviate this pain, it prints a warning on console and warns user timeline client could not be initialized because of dependency issue, more user friendly.

          The goad sounds reasonable but I don't think that justifies the behavior to catch and swallow an Error. What we can do is to clearly document this behavior as a known issue, suggest uses to try disable timeline services when seeing this error, instead of directly assume the root cause of an error?

          Show
          gtCarrera9 Li Lu added a comment - Thanks Weiwei Yang . The fix here was trying to alleviate this pain, it prints a warning on console and warns user timeline client could not be initialized because of dependency issue, more user friendly. The goad sounds reasonable but I don't think that justifies the behavior to catch and swallow an Error. What we can do is to clearly document this behavior as a known issue, suggest uses to try disable timeline services when seeing this error, instead of directly assume the root cause of an error?
          Hide
          cheersyang Weiwei Yang added a comment -

          Thanks Li Lu, hi Wei-Chiu Chuang what's your thought on this?

          Show
          cheersyang Weiwei Yang added a comment - Thanks Li Lu , hi Wei-Chiu Chuang what's your thought on this?
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Okay. I think it's a better idea to bail out after emitting an warning message that suggests setting hadoop.yarn.timeline-service.enabled=false to disable timeline service. How does that sound?

          I am going to revert this commit. Weiwei Yang can you work on a new patch based on Li Lu's suggestion?

          Show
          jojochuang Wei-Chiu Chuang added a comment - Okay. I think it's a better idea to bail out after emitting an warning message that suggests setting hadoop.yarn.timeline-service.enabled=false to disable timeline service. How does that sound? I am going to revert this commit. Weiwei Yang can you work on a new patch based on Li Lu 's suggestion?
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks Wei-Chiu Chuang! Let's not revert the change directly since the code base changed a lot since the commit. Weiwei Yang maybe you'd like to open a new JIRA and fix the issue there? Thanks!

          Show
          gtCarrera9 Li Lu added a comment - Thanks Wei-Chiu Chuang ! Let's not revert the change directly since the code base changed a lot since the commit. Weiwei Yang maybe you'd like to open a new JIRA and fix the issue there? Thanks!
          Hide
          cheersyang Weiwei Yang added a comment -

          Hi Wei-Chiu Chuang and Li Lu

          Sure, I opened YARN-6177 to get this fixed as the way Li Lu suggested. Lets move the discussion over to that ticket, and closed this one.

          Thank you.

          Show
          cheersyang Weiwei Yang added a comment - Hi Wei-Chiu Chuang and Li Lu Sure, I opened YARN-6177 to get this fixed as the way Li Lu suggested. Lets move the discussion over to that ticket, and closed this one. Thank you.

            People

            • Assignee:
              cheersyang Weiwei Yang
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              4 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development