Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7766

Introduce a new config property for YARN Service dependency tarball location

Details

    • Reviewed

    Description

      Introduce a new config property (something like yarn.service.framework.path in-line with mapreduce.application.framework.path) for YARN Service dependency tarball location. This will provide flexibility to the user/cluster-admin to upload the dependency tarball to a location of their choice. If this config property is not set, YARN Service client will default to uploading all dependency jars from the client-host's classpath for every service launch request (as it does today).

      Also, accept an optional destination HDFS location for -enableFastLaunch command, to specify the location where user/cluster-admin wants to upload the tarball. If not specified, let's default it to the location we use today. The cluster-admin still needs to set yarn.service.framework.path to this default location otherwise it will not be used. So the command-line will become something like this -

      yarn app -enableFastLaunch [<Destination Folder>]

      Attachments

        1. YARN-7766.001.patch
          19 kB
          Gour Saha
        2. YARN-7766.002.patch
          19 kB
          Gour Saha
        3. YARN-7766.003.patch
          16 kB
          Gour Saha
        4. YARN-7766.004.patch
          16 kB
          Gour Saha

        Activity

          gsaha Gour Saha added a comment -

          jianhe, please review the patch when you get a chance.

          gsaha Gour Saha added a comment - jianhe , please review the patch when you get a chance.
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 33s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 15m 21s trunk passed
          +1 compile 7m 23s trunk passed
          +1 checkstyle 1m 2s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 shadedclient 10m 58s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 17s trunk passed
          +1 javadoc 0m 49s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 49s the patch passed
          +1 compile 6m 21s the patch passed
          +1 javac 6m 21s the patch passed
          -0 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 347 unchanged - 2 fixed = 351 total (was 349)
          +1 mvnsite 1m 4s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 9m 25s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 23s the patch passed
          +1 javadoc 0m 41s the patch passed
                Other Tests
          -1 unit 27m 7s hadoop-yarn-client in the patch failed.
          +1 unit 3m 27s hadoop-yarn-services-core in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          90m 6s



          Reason Tests
          Failed junit tests hadoop.yarn.client.api.impl.TestAMRMClient



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue YARN-7766
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12906735/YARN-7766.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 3e8019a75483 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 37f4696
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19337/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/19337/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19337/testReport/
          Max. process+thread count 723 (vs. ulimit of 5000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/19337/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 15m 21s trunk passed +1 compile 7m 23s trunk passed +1 checkstyle 1m 2s trunk passed +1 mvnsite 1m 9s trunk passed +1 shadedclient 10m 58s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 17s trunk passed +1 javadoc 0m 49s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 49s the patch passed +1 compile 6m 21s the patch passed +1 javac 6m 21s the patch passed -0 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 347 unchanged - 2 fixed = 351 total (was 349) +1 mvnsite 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 9m 25s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 41s the patch passed       Other Tests -1 unit 27m 7s hadoop-yarn-client in the patch failed. +1 unit 3m 27s hadoop-yarn-services-core in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 90m 6s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestAMRMClient Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7766 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12906735/YARN-7766.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 3e8019a75483 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 37f4696 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19337/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/19337/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19337/testReport/ Max. process+thread count 723 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/19337/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          gsaha Gour Saha added a comment -

          Fixing 1 checkstyle issue with 002 patch. The other 3 adheres to the existing formatting in the class, so did not want to break it. The test failure is unrelated to this patch.

          gsaha Gour Saha added a comment - Fixing 1 checkstyle issue with 002 patch. The other 3 adheres to the existing formatting in the class, so did not want to break it. The test failure is unrelated to this patch.
          genericqa genericqa added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 49s Maven dependency ordering for branch
          +1 mvninstall 15m 16s trunk passed
          +1 compile 7m 35s trunk passed
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 shadedclient 11m 45s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 23s trunk passed
          +1 javadoc 0m 50s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 49s the patch passed
          +1 compile 6m 41s the patch passed
          +1 javac 6m 41s the patch passed
          -0 checkstyle 1m 1s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 345 unchanged - 2 fixed = 348 total (was 347)
          +1 mvnsite 1m 6s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 10m 27s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 34s the patch passed
          +1 javadoc 0m 48s the patch passed
                Other Tests
          +1 unit 27m 39s hadoop-yarn-client in the patch passed.
          +1 unit 3m 30s hadoop-yarn-services-core in the patch passed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          94m 2s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue YARN-7766
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12906772/YARN-7766.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 8a2b5335b2e8 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9e4f52d
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19346/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19346/testReport/
          Max. process+thread count 721 (vs. ulimit of 5000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/19346/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 49s Maven dependency ordering for branch +1 mvninstall 15m 16s trunk passed +1 compile 7m 35s trunk passed +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 10s trunk passed +1 shadedclient 11m 45s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 23s trunk passed +1 javadoc 0m 50s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 49s the patch passed +1 compile 6m 41s the patch passed +1 javac 6m 41s the patch passed -0 checkstyle 1m 1s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 345 unchanged - 2 fixed = 348 total (was 347) +1 mvnsite 1m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 10m 27s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 48s the patch passed       Other Tests +1 unit 27m 39s hadoop-yarn-client in the patch passed. +1 unit 3m 30s hadoop-yarn-services-core in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 94m 2s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7766 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12906772/YARN-7766.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 8a2b5335b2e8 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 9e4f52d maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19346/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19346/testReport/ Max. process+thread count 721 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/19346/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          jianhe Jian He added a comment -
            public int actionDependency(String destinationFolder, boolean overwrite)
                throws IOException, YarnException {
              String currentUser = RegistryUtils.currentUser();
              LOG.info("Running command as user {}", currentUser);
          
              Path dependencyLibTarGzip = fs.getDependencyTarGzip(true,
                  destinationFolder);
          
          • For above code, feel like we don't need to pass in a 'true' flag (this avoids the other chain of caller changes which passes in "false"). We can do special code handling right in here something like below ?
                if (destinationFolder == null) {
                  destinationFolder = String.format(YarnServiceConstants.DEPENDENCY_DIR,
                      VersionInfo.getVersion());
                }
                Path dependencyLibTarGzip = new Path(destinationFolder,
                    YarnServiceConstants.DEPENDENCY_TAR_GZ_FILE_NAME
                        + YarnServiceConstants.DEPENDENCY_TAR_GZ_FILE_EXT);
            
          jianhe Jian He added a comment - public int actionDependency( String destinationFolder, boolean overwrite) throws IOException, YarnException { String currentUser = RegistryUtils.currentUser(); LOG.info( "Running command as user {}" , currentUser); Path dependencyLibTarGzip = fs.getDependencyTarGzip( true , destinationFolder); For above code, feel like we don't need to pass in a 'true' flag (this avoids the other chain of caller changes which passes in "false"). We can do special code handling right in here something like below ? if (destinationFolder == null ) { destinationFolder = String .format(YarnServiceConstants.DEPENDENCY_DIR, VersionInfo.getVersion()); } Path dependencyLibTarGzip = new Path(destinationFolder, YarnServiceConstants.DEPENDENCY_TAR_GZ_FILE_NAME + YarnServiceConstants.DEPENDENCY_TAR_GZ_FILE_EXT);
          gsaha Gour Saha added a comment -

          jianhe, thank you for reviewing the patch. I incorporated your suggestion and uploaded 003 patch.

          gsaha Gour Saha added a comment - jianhe , thank you for reviewing the patch. I incorporated your suggestion and uploaded 003 patch.
          gsaha Gour Saha added a comment -

          jianhe, uploaded a 004 patch rebased to latest trunk and a console msg update.

          gsaha Gour Saha added a comment - jianhe , uploaded a 004 patch rebased to latest trunk and a console msg update.
          genericqa genericqa added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 9m 38s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 22m 31s Maven dependency ordering for branch
          +1 mvninstall 16m 37s trunk passed
          +1 compile 8m 5s trunk passed
          +1 checkstyle 1m 5s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 shadedclient 12m 26s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 20s trunk passed
          +1 javadoc 0m 44s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 50s the patch passed
          +1 compile 7m 22s the patch passed
          +1 javac 7m 22s the patch passed
          -0 checkstyle 1m 2s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 314 unchanged - 4 fixed = 318 total (was 318)
          +1 mvnsite 1m 2s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 11m 4s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 33s the patch passed
          +1 javadoc 0m 43s the patch passed
                Other Tests
          +1 unit 27m 35s hadoop-yarn-client in the patch passed.
          +1 unit 3m 37s hadoop-yarn-services-core in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          128m 34s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue YARN-7766
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12907295/YARN-7766.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 50120e47bf30 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6347b22
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19400/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19400/testReport/
          Max. process+thread count 694 (vs. ulimit of 5000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/19400/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 9m 38s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 22m 31s Maven dependency ordering for branch +1 mvninstall 16m 37s trunk passed +1 compile 8m 5s trunk passed +1 checkstyle 1m 5s trunk passed +1 mvnsite 1m 5s trunk passed +1 shadedclient 12m 26s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 20s trunk passed +1 javadoc 0m 44s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 50s the patch passed +1 compile 7m 22s the patch passed +1 javac 7m 22s the patch passed -0 checkstyle 1m 2s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 314 unchanged - 4 fixed = 318 total (was 318) +1 mvnsite 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 4s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 43s the patch passed       Other Tests +1 unit 27m 35s hadoop-yarn-client in the patch passed. +1 unit 3m 37s hadoop-yarn-services-core in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 128m 34s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7766 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12907295/YARN-7766.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 50120e47bf30 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 6347b22 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19400/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19400/testReport/ Max. process+thread count 694 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/19400/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          jianhe Jian He added a comment -

          Committed, thanks Gour!

          jianhe Jian He added a comment - Committed, thanks Gour!
          jianhe Jian He added a comment -

          I'm wondering if below makes sense to be a followup item.

          Currently, if user doesn't supply location when run yarn app -enableFastLaunch, the jars will be put under this location

          hdfs:///yarn-services/<versionId>/service-dep.tar.gz
          

          Since API server is embedded in RM, should RM look for this location too if "yarn.service.framework.path" is not specified ?

          And if "yarn.service.framework.path" is not specified and still the file doesn't exist at above default location, I think RM can try to upload the jars to above default location instead, currently RM is uploading the jars to the location defined by below code. This folder is per app and also inconsistent with CLI location.

            protected Path addJarResource(String serviceName,
                Map<String, LocalResource> localResources)
                throws IOException, SliderException {
              Path libPath = fs.buildClusterDirPath(serviceName);
          

          By doing this, the next time a submission request comes, RM doesn't need to upload the jars again.

          jianhe Jian He added a comment - I'm wondering if below makes sense to be a followup item. Currently, if user doesn't supply location when run yarn app -enableFastLaunch, the jars will be put under this location hdfs: ///yarn-services/<versionId>/service-dep.tar.gz Since API server is embedded in RM, should RM look for this location too if "yarn.service.framework.path" is not specified ? And if "yarn.service.framework.path" is not specified and still the file doesn't exist at above default location, I think RM can try to upload the jars to above default location instead, currently RM is uploading the jars to the location defined by below code. This folder is per app and also inconsistent with CLI location. protected Path addJarResource( String serviceName, Map< String , LocalResource> localResources) throws IOException, SliderException { Path libPath = fs.buildClusterDirPath(serviceName); By doing this, the next time a submission request comes, RM doesn't need to upload the jars again.
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13539 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13539/)
          YARN-7766. Introduce a new config property for YARN Service dependency (jianhe: rev a72cdcc47a1b3a4fcadbc9e12bd1bae2bbf04d83)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/CoreFileSystem.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/provider/tarball/TarballProviderService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/YarnServiceConf.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13539 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13539/ ) YARN-7766 . Introduce a new config property for YARN Service dependency (jianhe: rev a72cdcc47a1b3a4fcadbc9e12bd1bae2bbf04d83) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/CoreFileSystem.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/provider/tarball/TarballProviderService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/YarnServiceConf.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
          gsaha Gour Saha added a comment -

          jianhe, now that we have the config property to override the default, I think your followup suggestions totally make sense. I filed YARN-7799 on your behalf to capture this.

          gsaha Gour Saha added a comment - jianhe , now that we have the config property to override the default, I think your followup suggestions totally make sense. I filed YARN-7799 on your behalf to capture this.

          People

            gsaha Gour Saha
            gsaha Gour Saha
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: