Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6137

Yarn client implicitly invoke ATS client which accesses HDFS

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Yarn is implicitly trying to invoke ATS Client even though client does not need it. and ATSClient code is trying to access hdfs. Due to that service is hitting GSS exception.
      Yarnclient is implicitly creating ats client that tries to access Hdfs.
      All servers that use yarnclient cannot be expected to change to accommodate this behavior.

        Activity

        Hide
        gtCarrera9 Li Lu added a comment -

        This appears to be an ATS v1.5 only issue, but a bigger question is, why do we need a timeline client within YarnClientImpl? To me the only thing needed is to renew delegation token. If this reference is inevitable, can we avoid creating the client at service start of yarn client impl? We can lazily create the client only when we need to renew the token?

        Show
        gtCarrera9 Li Lu added a comment - This appears to be an ATS v1.5 only issue, but a bigger question is, why do we need a timeline client within YarnClientImpl? To me the only thing needed is to renew delegation token. If this reference is inevitable, can we avoid creating the client at service start of yarn client impl? We can lazily create the client only when we need to renew the token?
        Hide
        gtCarrera9 Li Lu added a comment -

        First patch to fix this issue. Note that we always start a timeline client when we start a YarnClientImpl when timeline service is enabled. In ATS v1.5, timeline client will check HDFS access upon service start, this requires the yarn client user to be authenticated when it's started. In fact, users only need this client to renew timeline tokens under secured environment, and it's totally fine to firstly start the client user process, authenticate it, and then renew the delegation token.

        So in this patch I'm delaying the start of the timeline client to the first time user needs a delegation token. For secured environments, this allows the parent process (running this client) to finish authentication after service start, then use the timeline client to renew tokens. One thing I'm not sure about is if yarn client itself should be thread safe. If this is the case I can add some synchronization for the time client initialization.

        Another change I made is to remove one unit test to check if YarnClient would catch an Error, and fails the test when we did not catch the Error. To me this does not appear to be a reasonable behavior. Since it blocks testing, I'm removing it.

        Show
        gtCarrera9 Li Lu added a comment - First patch to fix this issue. Note that we always start a timeline client when we start a YarnClientImpl when timeline service is enabled. In ATS v1.5, timeline client will check HDFS access upon service start, this requires the yarn client user to be authenticated when it's started. In fact, users only need this client to renew timeline tokens under secured environment, and it's totally fine to firstly start the client user process, authenticate it, and then renew the delegation token. So in this patch I'm delaying the start of the timeline client to the first time user needs a delegation token. For secured environments, this allows the parent process (running this client) to finish authentication after service start, then use the timeline client to renew tokens. One thing I'm not sure about is if yarn client itself should be thread safe. If this is the case I can add some synchronization for the time client initialization. Another change I made is to remove one unit test to check if YarnClient would catch an Error, and fails the test when we did not catch the Error. To me this does not appear to be a reasonable behavior. Since it blocks testing, I'm removing it.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 50s trunk passed
        +1 compile 0m 21s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 23s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 32s trunk passed
        +1 javadoc 0m 15s trunk passed
        +1 mvninstall 0m 20s the patch passed
        +1 compile 0m 17s the patch passed
        +1 javac 0m 17s the patch passed
        -0 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 7 new + 90 unchanged - 2 fixed = 97 total (was 92)
        +1 mvnsite 0m 20s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 38s the patch passed
        +1 javadoc 0m 12s the patch passed
        -1 unit 16m 51s hadoop-yarn-client in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        37m 0s



        Reason Tests
        Failed junit tests hadoop.yarn.client.api.impl.TestYarnClient



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6137
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850914/YARN-6137-trunk.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e1825e9e6e4e 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e023584
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14823/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14823/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14823/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14823/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 50s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 20s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed -0 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 7 new + 90 unchanged - 2 fixed = 97 total (was 92) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 12s the patch passed -1 unit 16m 51s hadoop-yarn-client in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 37m 0s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6137 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850914/YARN-6137-trunk.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e1825e9e6e4e 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e023584 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14823/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14823/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14823/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/14823/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for the patch! I think overall the approach is reasonable.

        One thing I'm not sure about is if yarn client itself should be thread safe.

        I think it's reasonable to add a bit of synchronization just around the part where we try to init the timeline client so we don't have more than one thread doing it.

        For this code, shouldn't we just be checking if the client isn't null? The enabled boolean check looks redundant to me.

            if (timelineServiceEnabled && (timelineClient != null)) {
        

        I didn't look into the unit test failure, but it looks related.

        Show
        jlowe Jason Lowe added a comment - Thanks for the patch! I think overall the approach is reasonable. One thing I'm not sure about is if yarn client itself should be thread safe. I think it's reasonable to add a bit of synchronization just around the part where we try to init the timeline client so we don't have more than one thread doing it. For this code, shouldn't we just be checking if the client isn't null? The enabled boolean check looks redundant to me. if (timelineServiceEnabled && (timelineClient != null )) { I didn't look into the unit test failure, but it looks related.
        Hide
        gtCarrera9 Li Lu added a comment -

        Thanks Jason Lowe for the review! A new patch to address all review comments.

        Show
        gtCarrera9 Li Lu added a comment - Thanks Jason Lowe for the review! A new patch to address all review comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 14m 13s trunk passed
        +1 compile 0m 25s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 26s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 40s trunk passed
        +1 javadoc 0m 17s trunk passed
        +1 mvninstall 0m 26s the patch passed
        +1 compile 0m 23s the patch passed
        +1 javac 0m 23s the patch passed
        -0 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 1 new + 81 unchanged - 11 fixed = 82 total (was 92)
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 49s the patch passed
        +1 javadoc 0m 15s the patch passed
        +1 unit 18m 1s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        39m 29s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6137
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851518/YARN-6137-trunk.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 77475e2b0c12 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a87e385
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14856/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14856/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14856/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 13s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 40s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 26s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed -0 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 1 new + 81 unchanged - 11 fixed = 82 total (was 92) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 49s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 18m 1s hadoop-yarn-client in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 39m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6137 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851518/YARN-6137-trunk.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 77475e2b0c12 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a87e385 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14856/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14856/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/14856/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        +1 lgtm. Committing this.

        Show
        jlowe Jason Lowe added a comment - +1 lgtm. Committing this.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Li Lu! I committed this to trunk, branch-2, and branch-2.8.

        Show
        jlowe Jason Lowe added a comment - Thanks, Li Lu ! I committed this to trunk, branch-2, and branch-2.8.
        Hide
        gtCarrera9 Li Lu added a comment -

        Thanks Jason Lowe for the review and commit!

        Show
        gtCarrera9 Li Lu added a comment - Thanks Jason Lowe for the review and commit!
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #11223 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11223/)
        YARN-6137. Yarn client implicitly invoke ATS client which accesses HDFS. (jlowe: rev 37b4acf7cee1f05599a84bbb1ebf07979a71f82f)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #11223 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11223/ ) YARN-6137 . Yarn client implicitly invoke ATS client which accesses HDFS. (jlowe: rev 37b4acf7cee1f05599a84bbb1ebf07979a71f82f) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          People

          • Assignee:
            gtCarrera9 Li Lu
            Reporter:
            yeshavora Yesha Vora
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development