Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7799

YARN Service dependency follow up work

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0, 3.1.1
    • Component/s: client, resourcemanager
    • Labels:
      None

      Description

      As per Jian He these are some followup items that make sense to do after YARN-7766. Quoting Jian's comment below -

      Currently, if user doesn't supply location when run yarn app -enableFastLaunch, the jars will be put under this location

      hdfs:///yarn-services/<versionId>/service-dep.tar.gz
      

      Since API server is embedded in RM, should RM look for this location too if "yarn.service.framework.path" is not specified ?

      And if "yarn.service.framework.path" is not specified and still the file doesn't exist at above default location, I think RM can try to upload the jars to above default location instead, currently RM is uploading the jars to the location defined by below code. This folder is per app and also inconsistent with CLI location.

        protected Path addJarResource(String serviceName,
            Map<String, LocalResource> localResources)
            throws IOException, SliderException {
          Path libPath = fs.buildClusterDirPath(serviceName);
      

      By doing this, the next time a submission request comes, RM doesn't need to upload the jars again.

        Attachments

        1. YARN-7799.1.patch
          10 kB
          Billie Rinaldi
        2. YARN-7799.2.patch
          18 kB
          Billie Rinaldi
        3. YARN-7799.3.patch
          19 kB
          Billie Rinaldi
        4. YARN-7799.4.patch
          21 kB
          Billie Rinaldi
        5. YARN-7799.5.patch
          21 kB
          Billie Rinaldi

          Activity

            People

            • Assignee:
              billie.rinaldi Billie Rinaldi
              Reporter:
              gsaha Gour Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: