Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4421

Run MapReduce framework via the distributed cache

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 3.0.0, 2.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      Currently MR AM depends on MR jars being deployed on all nodes via implicit dependency on YARN_APPLICATION_CLASSPATH.

      We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and, probably, just rely on adding a shaded MR jar along with job.jar to the dist-cache.

      1. MAPREDUCE-4421-4.patch
        21 kB
        Jason Lowe
      2. MAPREDUCE-4421-3.patch
        21 kB
        Jason Lowe
      3. MAPREDUCE-4421-2.patch
        20 kB
        Jason Lowe
      4. MAPREDUCE-4421.patch
        12 kB
        Jason Lowe
      5. MAPREDUCE-4421.patch
        12 kB
        Jason Lowe

        Issue Links

          Activity

          Hide
          Arun C Murthy added a comment -

          In particular we should remove $YARN_HOME/share/hadoop/mapreduce/* and $YARN_HOME/share/hadoop/mapreduce/lib/* from YARN_APPLICATION_CLASSPATH.

          Show
          Arun C Murthy added a comment - In particular we should remove $YARN_HOME/share/hadoop/mapreduce/* and $YARN_HOME/share/hadoop/mapreduce/lib/* from YARN_APPLICATION_CLASSPATH.
          Hide
          Arun C Murthy added a comment -

          Forgot to add - the major advantage of doing this is that it helps deliver the promise of multiple MR versions in same YARN cluster.

          Show
          Arun C Murthy added a comment - Forgot to add - the major advantage of doing this is that it helps deliver the promise of multiple MR versions in same YARN cluster.
          Hide
          Harsh J added a comment -

          Good idea. How do we handle the native lib dependencies though? Do we ship that as well, or keep a static resource at the NM? Different MR versions may end up needing different native lib sets, perhaps?

          Show
          Harsh J added a comment - Good idea. How do we handle the native lib dependencies though? Do we ship that as well, or keep a static resource at the NM? Different MR versions may end up needing different native lib sets, perhaps?
          Hide
          Kihwal Lee added a comment -

          How do we handle the native lib dependencies though? Do we ship that as well, or keep a static resource at the NM?

          There are two parts to this issue:

          1. Specifying dependency: App-level dependencies on native libs should be specified within the app, not by YARN. Apps can also allow each job to specify additional dependencies. A proper merging of LD_LIBRARY_PATH from job, app and hadoop must be done. (was -Djava.library.path in mrv1) Who merges what needs to be made clear (a section in the app writer's guide?).
          2. Making libs available: The simplest way is to let the app ship them for each job. But admins may choose to host app-level dependencies (in a multiversion aware manner) and even some of popular job-level dependencies. YARN should never automatically shove everything to apps. There are pros and cons in both approaches. A well designed app will support both.

          YARN should not remove or override legitimate app/job-level dependencies. I think YARN already satisfies this, but there might be some areas that need improvement. We should also provide a clear guide on how to manage dependencies for app writers and admins.

          This is quite similar to jar dependency management (this jira) in principle.

          Show
          Kihwal Lee added a comment - How do we handle the native lib dependencies though? Do we ship that as well, or keep a static resource at the NM? There are two parts to this issue: Specifying dependency: App-level dependencies on native libs should be specified within the app, not by YARN. Apps can also allow each job to specify additional dependencies. A proper merging of LD_LIBRARY_PATH from job, app and hadoop must be done. (was -Djava.library.path in mrv1) Who merges what needs to be made clear (a section in the app writer's guide?). Making libs available: The simplest way is to let the app ship them for each job. But admins may choose to host app-level dependencies (in a multiversion aware manner) and even some of popular job-level dependencies. YARN should never automatically shove everything to apps. There are pros and cons in both approaches. A well designed app will support both. YARN should not remove or override legitimate app/job-level dependencies. I think YARN already satisfies this, but there might be some areas that need improvement. We should also provide a clear guide on how to manage dependencies for app writers and admins. This is quite similar to jar dependency management (this jira) in principle.
          Hide
          Robert Joseph Evans added a comment -

          I would say too that we want to have these jars be something that can be configured to already be in HDFS. When I looked at our distributed cache recently more then 50% of the files in the cache were all copies of the same pig jar. If we start shipping MR jars we really should have it so that they are cache friendly.

          Show
          Robert Joseph Evans added a comment - I would say too that we want to have these jars be something that can be configured to already be in HDFS. When I looked at our distributed cache recently more then 50% of the files in the cache were all copies of the same pig jar. If we start shipping MR jars we really should have it so that they are cache friendly.
          Hide
          Arun C Murthy added a comment -

          Bobby - if I wasn't clear, apologies.

          However yes, the ability for MR apps to reference an existing HDFS file (jar) via dist-cache is a very important requirement.

          Show
          Arun C Murthy added a comment - Bobby - if I wasn't clear, apologies. However yes, the ability for MR apps to reference an existing HDFS file (jar) via dist-cache is a very important requirement.
          Hide
          Arun C Murthy added a comment -

          How do we handle the native lib dependencies though? Do we ship that as well, or keep a static resource at the NM?

          Yep, we ship those via dist-cache too.

          Show
          Arun C Murthy added a comment - How do we handle the native lib dependencies though? Do we ship that as well, or keep a static resource at the NM? Yep, we ship those via dist-cache too.
          Hide
          Harsh J added a comment -

          Thanks Kihwal and Arun. I guess if its via a central, i.e., a dist-cache, we may lose some heterogenous cluster abilities (different OS versions forming a single cluster, with a chance that one's natives may not work on the other and needs uniquely compiled sets for each).

          Not much of a worry, and I've hardly seen folks do this in the long run, but might be worth providing a solution for, in the future?

          Show
          Harsh J added a comment - Thanks Kihwal and Arun. I guess if its via a central, i.e., a dist-cache, we may lose some heterogenous cluster abilities (different OS versions forming a single cluster, with a chance that one's natives may not work on the other and needs uniquely compiled sets for each). Not much of a worry, and I've hardly seen folks do this in the long run, but might be worth providing a solution for, in the future?
          Hide
          Alejandro Abdelnur added a comment -

          This seems more a general YARN AM issue than a MR issue.

          How about tweaking YARN so a client can specify a yarn.am.lib.path property, which would be a FileSystem accessible to all NMs (typically HDFS)? Then YARN would use that similarly to distributed cache entries, localizing all JARs before bootstrapping the AM & tasks.

          Show
          Alejandro Abdelnur added a comment - This seems more a general YARN AM issue than a MR issue. How about tweaking YARN so a client can specify a yarn.am.lib.path property, which would be a FileSystem accessible to all NMs (typically HDFS)? Then YARN would use that similarly to distributed cache entries, localizing all JARs before bootstrapping the AM & tasks.
          Hide
          Arun C Murthy added a comment -

          Alejandro, instead of adding new properties, we can just use the dist-cache. Thoughts?

          Show
          Arun C Murthy added a comment - Alejandro, instead of adding new properties, we can just use the dist-cache. Thoughts?
          Hide
          Robert Joseph Evans added a comment -

          I think the distcache should be fine, but we are still going to need a new property so that MR knows where in HDFS the corresponding jars have been installed. Preferably it should also have the version number somewhere in there so if someone builds a new experimental version of hadoop with some modifications they don't accidentally get the wrong version.

          Show
          Robert Joseph Evans added a comment - I think the distcache should be fine, but we are still going to need a new property so that MR knows where in HDFS the corresponding jars have been installed. Preferably it should also have the version number somewhere in there so if someone builds a new experimental version of hadoop with some modifications they don't accidentally get the wrong version.
          Hide
          Alejandro Abdelnur added a comment -

          Let me explain in more detail what I was thinking (along the lines of Oozie's sharelib for actions):

          1 OPS maintains a /user/am/share/lib/ directory under this directory you have a sub-directory for each AM, ie: mapred, distributedshell

          2 yarn-site.xml (or an am-site.xml) defines the following properties:

          yarn.am.lib=/user/am/share/lib
          mapred.am.lib=${yarn.am.lib}/mapred
          distributedshell.am.lib=${yarn.am.lib}/distributedshell
          

          3 AM client injects to the configuration, via DistributedCache, all the JARs in the directory of the corresponding ####.am.lib property.

          Note that this mechanism allows to support multiple versions of AMs, to OPs to define the default version, to even fix it with final, and to users to use alternate ones.

          Show
          Alejandro Abdelnur added a comment - Let me explain in more detail what I was thinking (along the lines of Oozie's sharelib for actions): 1 OPS maintains a /user/am/share/lib/ directory under this directory you have a sub-directory for each AM, ie: mapred , distributedshell 2 yarn-site.xml (or an am-site.xml ) defines the following properties: yarn.am.lib=/user/am/share/lib mapred.am.lib=${yarn.am.lib}/mapred distributedshell.am.lib=${yarn.am.lib}/distributedshell 3 AM client injects to the configuration, via DistributedCache, all the JARs in the directory of the corresponding ####.am.lib property. Note that this mechanism allows to support multiple versions of AMs, to OPs to define the default version, to even fix it with final, and to users to use alternate ones.
          Hide
          Arun C Murthy added a comment -

          Tucu - I think we are close, but I don't want MR AM or DistShell AM configs in yarn-site.xml. They belong in mapred-site.xml or distshell-site.xml etc. Makes sense?

          Show
          Arun C Murthy added a comment - Tucu - I think we are close, but I don't want MR AM or DistShell AM configs in yarn-site.xml. They belong in mapred-site.xml or distshell-site.xml etc. Makes sense?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          With almost all of the cleanup/separation of YARN and MapReduce out of the way, we are now in a position to move this forward.

          Going to take a shot at it, assigning this to myself.

          I'll start with the distribution of jars, will split the issue of handling native libs out into another follow-up ticket.

          Show
          Vinod Kumar Vavilapalli added a comment - With almost all of the cleanup/separation of YARN and MapReduce out of the way, we are now in a position to move this forward. Going to take a shot at it, assigning this to myself. I'll start with the distribution of jars, will split the issue of handling native libs out into another follow-up ticket.
          Hide
          Alejandro Abdelnur added a comment -

          Arun - make sense, AMs configs should not pollute YARN config.

          Now building on this ...

          We are aiming at a YARN deployment which does not know anything about AMs that run on it. This means the cluster setup should not have AMs site.xml files deployed in it. This means that the cluster setup should not have AMs JARs deployed in it.

          For one off custom AMs, JARs and config would be all provided by the client on submission.

          For AMs like MapReduce, DistributedShell and widely used AMs in a given cluster, their JARs and config site.xml files would be in HDFS. Then the 'yarn' client script should support a '-ampath=' (HDFS path to AM resources) option and/or a '-amname=' (logical name of the AM resources, a new config file am-site.xml in the cluster would have this mapping for blessed AMs, as suggested in my prev comment).

          Thoughts?

          Show
          Alejandro Abdelnur added a comment - Arun - make sense, AMs configs should not pollute YARN config. Now building on this ... We are aiming at a YARN deployment which does not know anything about AMs that run on it. This means the cluster setup should not have AMs site.xml files deployed in it. This means that the cluster setup should not have AMs JARs deployed in it. For one off custom AMs, JARs and config would be all provided by the client on submission. For AMs like MapReduce, DistributedShell and widely used AMs in a given cluster, their JARs and config site.xml files would be in HDFS. Then the 'yarn' client script should support a '-ampath=' (HDFS path to AM resources) option and/or a '-amname=' (logical name of the AM resources, a new config file am-site.xml in the cluster would have this mapping for blessed AMs, as suggested in my prev comment). Thoughts?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This means the cluster setup should not have AMs site.xml files deployed in it.

          Then the 'yarn' client script should support a '-ampath=' (HDFS path to AM resources) option and/or a '-amname=' (logical name of the AM resources, a new config file am-site.xml in the cluster would have this mapping for blessed AMs, as suggested in my prev comment).

          That is possible today itself with the separate YARN_CONF_DIR. I haven't tested with separate conf-dirs but can check right away. Essentially, MR has its conf file (mapred-default.xml), dist-shell could have its own. We can argue either ways about creating a new config file am-site.xml for 'blessed' AMs.

          This means that the cluster setup should not have AMs JARs deployed in it.

          This is already the case. I have the test-cluster with HADOOP_YARN_HOME ahd HADOOP_MAPRED_HOME separate. So yeah, YARN doesn't have any mapred jars (except the shuffle related ones, which is not for the AMs)

          For one off custom AMs, JARs and config would be all provided by the client on submission.

          For AMs like MapReduce, DistributedShell and widely used AMs in a given cluster, their JARs and config site.xml files would be in HDFS.

          Yarn doesn't care how the AM related jars are managed. All it needs to know at the end of the day is a FS location of all the jars needed by the app. So the jars can be managed in two ways. The framework specific clients can pick up the AM jars

          • from a public location on DFS and populate dist-cache
          • or a local installation on the client and upload it to a private location on DFS and populate the dist-cache
          • or if the AM jars happen to be installed on every node, construct the classpath referring to those jars.

          Today MR AM implements the third option above, this JIRA is to enable the first two options.

          Show
          Vinod Kumar Vavilapalli added a comment - This means the cluster setup should not have AMs site.xml files deployed in it. Then the 'yarn' client script should support a '-ampath=' (HDFS path to AM resources) option and/or a '-amname=' (logical name of the AM resources, a new config file am-site.xml in the cluster would have this mapping for blessed AMs, as suggested in my prev comment). That is possible today itself with the separate YARN_CONF_DIR. I haven't tested with separate conf-dirs but can check right away. Essentially, MR has its conf file (mapred-default.xml), dist-shell could have its own. We can argue either ways about creating a new config file am-site.xml for 'blessed' AMs. This means that the cluster setup should not have AMs JARs deployed in it. This is already the case. I have the test-cluster with HADOOP_YARN_HOME ahd HADOOP_MAPRED_HOME separate. So yeah, YARN doesn't have any mapred jars (except the shuffle related ones, which is not for the AMs) For one off custom AMs, JARs and config would be all provided by the client on submission. For AMs like MapReduce, DistributedShell and widely used AMs in a given cluster, their JARs and config site.xml files would be in HDFS. Yarn doesn't care how the AM related jars are managed. All it needs to know at the end of the day is a FS location of all the jars needed by the app. So the jars can be managed in two ways. The framework specific clients can pick up the AM jars from a public location on DFS and populate dist-cache or a local installation on the client and upload it to a private location on DFS and populate the dist-cache or if the AM jars happen to be installed on every node, construct the classpath referring to those jars. Today MR AM implements the third option above, this JIRA is to enable the first two options.
          Hide
          Jason Lowe added a comment -

          Submitting a patch to try to move this forward. We're very interested in the ability to patch issues in the MapReduce framework without having to bring down the cluster and/or push a new version to all nodes.

          This patch adds a new config, mapreduce.application.framework.path, which defaults to being unset. If set, it specifies a path to an archive containing the MR framework to use with the job. Normally this would point to a public location within HDFS, and the archive would contain all the MR jars and their dependencies, i.e.: MR jars, YARN client jars, HDFS client, common, and all their dependencies.

          This allows ops to deposit a single archive into HDFS that contains the MR framework and configure mapred-site.xml to use it. That framework is then lazily deployed to the nodes. A new version can be uploaded to another path, the mapred-site.xml updated, and then all future jobs run with the new version while all currently running jobs proceed with the previous version. Or ops can avoid pushing the mapred-site.xml change out to all gateway/launcher boxes by using a standard path symlink that always points to the current version to use. New versions can be deployed, the symlink moved to them, and jobs implicitly pick up the new version without pushing a corresponding mapred-site.xml change.

          I've tested this by taking the entire hadoop-3.0.0-SNAPSHOT.tar.gz file and placing it in HDFS under /mapred/. Admittedly, this is not the most efficient deployment, but it does include everything necessary. I then set mapreduce.application.framework.path to /mapred/hadoop-3.0.0-SNAPSHOT.tar.gz#mr-framework and mapreduce.application.classpath to:

          $PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
          

          The job then ran with my specified version of the MR framework instead of the one deployed to the nodes. The application classpath is complicated because I used the standard distribution tarball. I could have easily built a custom tarball with all the jars at the top directory and simply had a classpath of:

          $PWD/mr-framework/*.jar
          

          The framework is lazily deployed via the distributed cache, so nodes take a localization hit the first time they see a job with a specified framework path. However subsequent jobs with the same framework run quickly, and I saw no performance difference between jobs using a custom framework and jobs using the cluster-installed framework on nodes that had already localized the specified framework.

          Note that there is still a dependency on deployed MR jars with respect to the shuffle service running on all the nodes. With this patch, new MR versions can only be used when the old shuffle service on all nodes is compatible with the new version. Fixing this requires the ability to specify auxiliary services with YARN application submissions and have those lazily deploy to nodes that are allocated for the application. (And ideally subsequently refcounted and retired once no longer necessary.)

          Show
          Jason Lowe added a comment - Submitting a patch to try to move this forward. We're very interested in the ability to patch issues in the MapReduce framework without having to bring down the cluster and/or push a new version to all nodes. This patch adds a new config, mapreduce.application.framework.path , which defaults to being unset. If set, it specifies a path to an archive containing the MR framework to use with the job. Normally this would point to a public location within HDFS, and the archive would contain all the MR jars and their dependencies, i.e.: MR jars, YARN client jars, HDFS client, common, and all their dependencies. This allows ops to deposit a single archive into HDFS that contains the MR framework and configure mapred-site.xml to use it. That framework is then lazily deployed to the nodes. A new version can be uploaded to another path, the mapred-site.xml updated, and then all future jobs run with the new version while all currently running jobs proceed with the previous version. Or ops can avoid pushing the mapred-site.xml change out to all gateway/launcher boxes by using a standard path symlink that always points to the current version to use. New versions can be deployed, the symlink moved to them, and jobs implicitly pick up the new version without pushing a corresponding mapred-site.xml change. I've tested this by taking the entire hadoop-3.0.0-SNAPSHOT.tar.gz file and placing it in HDFS under /mapred/. Admittedly, this is not the most efficient deployment, but it does include everything necessary. I then set mapreduce.application.framework.path to /mapred/hadoop-3.0.0-SNAPSHOT.tar.gz#mr-framework and mapreduce.application.classpath to: $PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/* The job then ran with my specified version of the MR framework instead of the one deployed to the nodes. The application classpath is complicated because I used the standard distribution tarball. I could have easily built a custom tarball with all the jars at the top directory and simply had a classpath of: $PWD/mr-framework/*.jar The framework is lazily deployed via the distributed cache, so nodes take a localization hit the first time they see a job with a specified framework path. However subsequent jobs with the same framework run quickly, and I saw no performance difference between jobs using a custom framework and jobs using the cluster-installed framework on nodes that had already localized the specified framework. Note that there is still a dependency on deployed MR jars with respect to the shuffle service running on all the nodes. With this patch, new MR versions can only be used when the old shuffle service on all nodes is compatible with the new version. Fixing this requires the ability to specify auxiliary services with YARN application submissions and have those lazily deploy to nodes that are allocated for the application. (And ideally subsequently refcounted and retired once no longer necessary.)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594215/MAPREDUCE-4421.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1150 warnings).

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3899//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3899//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3899//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594215/MAPREDUCE-4421.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1150 warnings). +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3899//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3899//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3899//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Updated patch to fix extra warnings.

          Show
          Jason Lowe added a comment - Updated patch to fix extra warnings.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12594236/MAPREDUCE-4421.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3902//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3902//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12594236/MAPREDUCE-4421.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3902//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3902//console This message is automatically generated.
          Hide
          Sandy Ryza added a comment -

          Very cool. Would it make sense to allow directories as well in mapreduce.application.framework.path? That would make it easier to swap out a jar without rebuilding the tarball. Also, what would it take to make this work easily for non-MR frameworks?

          Does the distributed cache actually cache things in between jobs? The javadoc says "Its efficiency stems from the fact that the files are only copied once per job and the ability to cache archives which are un-archived on the slaves." If so, we should probably modify the javadoc.

          Show
          Sandy Ryza added a comment - Very cool. Would it make sense to allow directories as well in mapreduce.application.framework.path? That would make it easier to swap out a jar without rebuilding the tarball. Also, what would it take to make this work easily for non-MR frameworks? Does the distributed cache actually cache things in between jobs? The javadoc says "Its efficiency stems from the fact that the files are only copied once per job and the ability to cache archives which are un-archived on the slaves." If so, we should probably modify the javadoc.
          Hide
          Jason Lowe added a comment -

          Would it make sense to allow directories as well in mapreduce.application.framework.path? That would make it easier to swap out a jar without rebuilding the tarball.

          The problem with directories is that officially they are unsupported in the distributed cache. Besides that, from a practical standpoint, it's much more difficult for a nodemanager to verify it doesn't need to localize anything when the item being localized is an arbitrary directory tree. That's a lot of HDFS stats to do vs. just one for the archive case.

          Does the distributed cache actually cache things in between jobs?

          Yes, it does if it can. It depends upon the visibility of the item being localized. If it's PUBLIC the resource will be cached and reused among all users and all jobs. If PRIVATE the resource will be cached only per-user but reused between jobs for that user. If APPLICATION then it will only be localized for a single job. See LocalResourceVisibility and ClientDistributedCacheManager.determineCacheVisibilities for some details.

          The javadoc is correct in that even for the APPLICATION case a resource will only be localized once even though multiple containers may run on the same node, so it's more efficient than just letting the tasks hit HDFS directly for the resource when multiple tasks run on the same node and the resource is needed by all tasks.

          Show
          Jason Lowe added a comment - Would it make sense to allow directories as well in mapreduce.application.framework.path? That would make it easier to swap out a jar without rebuilding the tarball. The problem with directories is that officially they are unsupported in the distributed cache. Besides that, from a practical standpoint, it's much more difficult for a nodemanager to verify it doesn't need to localize anything when the item being localized is an arbitrary directory tree. That's a lot of HDFS stats to do vs. just one for the archive case. Does the distributed cache actually cache things in between jobs? Yes, it does if it can. It depends upon the visibility of the item being localized. If it's PUBLIC the resource will be cached and reused among all users and all jobs. If PRIVATE the resource will be cached only per-user but reused between jobs for that user. If APPLICATION then it will only be localized for a single job. See LocalResourceVisibility and ClientDistributedCacheManager.determineCacheVisibilities for some details. The javadoc is correct in that even for the APPLICATION case a resource will only be localized once even though multiple containers may run on the same node, so it's more efficient than just letting the tasks hit HDFS directly for the resource when multiple tasks run on the same node and the resource is needed by all tasks.
          Hide
          Jason Lowe added a comment -

          Also, what would it take to make this work easily for non-MR frameworks?

          Other frameworks can do a similar trick, and note that I didn't have to make any YARN changes for it to work. Well, there is the aux service issue as I mentioned, but otherwise it can be done in a similar fashion. All it's basically doing from a YARN standpoint is having the client automatically bundle an archive as a LocalResource and doctoring the container environment accordingly. I thought I heard Tez was being deployed this way, but I haven't verified that.

          At the last Hadoop Summit, Alejandro Abdelnur had what I thought was a brilliant idea. Not only the idea of grabbing the framework support code for containers via HDFS, but having the client code come from an HDFS blob as well. There would be some yarn command to launch an application for a particular version of a framework, and that command would look in a configured place where frameworks are stored, pick out the appropriate version of the named framework, download the client code, and invoke the client to complete the rest of the app submission. The client could then bundle the rest of the framework in a similar fashion to how it's being done for MapReduce here.

          In essence, it would be a one-step deploy for app frameworks on YARN. Drop a blob in HDFS, and suddenly users can start using that framework even though they don't have any of the framework code installed at the time. There's still some big issues to work out, e.g.: how to download the client code efficiently (it becomes much like a localization issue with managing a cache of clients already downloaded, etc.), and I'm sure there's plenty of other devils in the details. But if accomplished, this would allow one-step deploys for application frameworks in YARN which I think would be a great feature.

          Show
          Jason Lowe added a comment - Also, what would it take to make this work easily for non-MR frameworks? Other frameworks can do a similar trick, and note that I didn't have to make any YARN changes for it to work. Well, there is the aux service issue as I mentioned, but otherwise it can be done in a similar fashion. All it's basically doing from a YARN standpoint is having the client automatically bundle an archive as a LocalResource and doctoring the container environment accordingly. I thought I heard Tez was being deployed this way, but I haven't verified that. At the last Hadoop Summit, Alejandro Abdelnur had what I thought was a brilliant idea. Not only the idea of grabbing the framework support code for containers via HDFS, but having the client code come from an HDFS blob as well. There would be some yarn command to launch an application for a particular version of a framework, and that command would look in a configured place where frameworks are stored, pick out the appropriate version of the named framework, download the client code, and invoke the client to complete the rest of the app submission. The client could then bundle the rest of the framework in a similar fashion to how it's being done for MapReduce here. In essence, it would be a one-step deploy for app frameworks on YARN. Drop a blob in HDFS, and suddenly users can start using that framework even though they don't have any of the framework code installed at the time. There's still some big issues to work out, e.g.: how to download the client code efficiently (it becomes much like a localization issue with managing a cache of clients already downloaded, etc.), and I'm sure there's plenty of other devils in the details. But if accomplished, this would allow one-step deploys for application frameworks in YARN which I think would be a great feature.
          Hide
          Sandy Ryza added a comment -

          Yes, it does if it can. It depends upon the visibility of the item being localized. If it's PUBLIC the resource will be cached and reused among all users and all jobs. If PRIVATE the resource will be cached only per-user but reused between jobs for that user. If APPLICATION then it will only be localized for a single job. See LocalResourceVisibility and ClientDistributedCacheManager.determineCacheVisibilities for some details.

          Ok awesome.

          The javadoc is correct in that even for the APPLICATION...

          Makes sense. Could still be a little more clear, but I'll take my complaints somewhere else.

          I think my only other question is about the config name. Should it not be mapreduce.job.framework.path to make it consistent with other MR configs?

          Show
          Sandy Ryza added a comment - Yes, it does if it can. It depends upon the visibility of the item being localized. If it's PUBLIC the resource will be cached and reused among all users and all jobs. If PRIVATE the resource will be cached only per-user but reused between jobs for that user. If APPLICATION then it will only be localized for a single job. See LocalResourceVisibility and ClientDistributedCacheManager.determineCacheVisibilities for some details. Ok awesome. The javadoc is correct in that even for the APPLICATION... Makes sense. Could still be a little more clear, but I'll take my complaints somewhere else. I think my only other question is about the config name. Should it not be mapreduce.job.framework.path to make it consistent with other MR configs?
          Hide
          Jason Lowe added a comment -

          Should it not be mapreduce.job.framework.path to make it consistent with other MR configs?

          I was trying to make it consistent with mapreduce.application.classpath since changing the framework config requires changing the classpath config. However if the consensus is the name should be different, I'm fine with that too.

          Show
          Jason Lowe added a comment - Should it not be mapreduce.job.framework.path to make it consistent with other MR configs? I was trying to make it consistent with mapreduce.application.classpath since changing the framework config requires changing the classpath config. However if the consensus is the name should be different, I'm fine with that too.
          Hide
          Sandy Ryza added a comment -

          My opinion is that mapreduce.application.classpath is also kind of a weird name. Is there anything about it that's different than the other configs that use "job"?

          Show
          Sandy Ryza added a comment - My opinion is that mapreduce.application.classpath is also kind of a weird name. Is there anything about it that's different than the other configs that use "job"?
          Hide
          Jason Lowe added a comment -

          I don't know the full history behind mapreduce.application.classpath, but my impression is that originally the property was not intended to be modified per-job but rather set as a cluster-wide property by ops since it effectively contains the location of the MR jars on each node. mapreduce.application.framework.path is similar, in the sense that I'd expect the common case would be ops to setup and maintain the MR framework versions located in HDFS and most users would simply use it as-is.

          Show
          Jason Lowe added a comment - I don't know the full history behind mapreduce.application.classpath, but my impression is that originally the property was not intended to be modified per-job but rather set as a cluster-wide property by ops since it effectively contains the location of the MR jars on each node. mapreduce.application.framework.path is similar, in the sense that I'd expect the common case would be ops to setup and maintain the MR framework versions located in HDFS and most users would simply use it as-is.
          Hide
          Hitesh Shah added a comment -

          Jason Lowe Had a few questions/comments related to the implementation/patch:

          • Why does classpath need to include all of common, hdfs and yarn jar locations? Assuming that MR is running on a YARN-based cluster, shouldn't the location of the core dependencies come from the cluster deployment i.e. via the env that the NM sets for a container. I believe the only jars that MR should have in its uploaded tarball should be the client jars. I understand that there is no clear boundary for client-side only jars for common and hdfs today ( for For YARN, I believe it should be simple to split out the client-side requirements ) but it is something we should aim for or assume that the jars deployed on the cluster are compatible.
          • I guess the underlying question is why use the full hadoop tarball and not just the mapreduce-only tarball? If MR is trully a user-land library, it should be treated as such and have a separate deployment approach.
          • I would vote to make the tar-ball in HDFS be the only way to run MR on YARN. Obviously, this cannot be done for 2.x but we should move to this model on trunk and not support the current approach at all there. Comments?
          • The other point is related to configs. Configuration still loads mapred-site and mapred-default files and new Configuration objects are created on the cluster. Are these files still expected on the cluster? job.xml does override these but cluster configs could still have final params. If this is meant to be addressed in a follow-up jira to ensure all MR configs come from the client, you can ignore this point for now.
          • How do you see framework name extracted from the path to be used? Is it just a safety check to ensure that it is found in the classpath? Will it have any relation to a version? A minor nit - framework name seems confusing in relation to the framework name in use from earlier i.e yarn vs local framework.
          • Description in the default-xml for mapreduce.application.framework.path does not mention the need for the URI fragment and how the fragment is used as a sanity check to the classpath.
          • Regarding versions, it seems like users will need to do 2 things. Change the location of the tarball on HDFS and modify the classpath. Users will need to know the exact structure of the classpath. In such a scenario, do defaults even make sense? On the other hand, if we define a common standard i.e. a base path for all MR tarballs, with each tarball in a defined structure ( possibly with version info added on later on for the code to infer the structure of the tarball ), all the user would need to do is specify the base path ( which could have a default value ) and a version which again has a default value. The latter approach would require the code to construct the necessary classpath if the upload path is in use. Do you have any comments on which of the 2 approaches makes more sense? The former is way more flexible but a bit more complex. The latter brittle/inflexible with respect to changing tarball structures but likely more easier to enforce a standard on.
          Show
          Hitesh Shah added a comment - Jason Lowe Had a few questions/comments related to the implementation/patch: Why does classpath need to include all of common, hdfs and yarn jar locations? Assuming that MR is running on a YARN-based cluster, shouldn't the location of the core dependencies come from the cluster deployment i.e. via the env that the NM sets for a container. I believe the only jars that MR should have in its uploaded tarball should be the client jars. I understand that there is no clear boundary for client-side only jars for common and hdfs today ( for For YARN, I believe it should be simple to split out the client-side requirements ) but it is something we should aim for or assume that the jars deployed on the cluster are compatible. I guess the underlying question is why use the full hadoop tarball and not just the mapreduce-only tarball? If MR is trully a user-land library, it should be treated as such and have a separate deployment approach. I would vote to make the tar-ball in HDFS be the only way to run MR on YARN. Obviously, this cannot be done for 2.x but we should move to this model on trunk and not support the current approach at all there. Comments? The other point is related to configs. Configuration still loads mapred-site and mapred-default files and new Configuration objects are created on the cluster. Are these files still expected on the cluster? job.xml does override these but cluster configs could still have final params. If this is meant to be addressed in a follow-up jira to ensure all MR configs come from the client, you can ignore this point for now. How do you see framework name extracted from the path to be used? Is it just a safety check to ensure that it is found in the classpath? Will it have any relation to a version? A minor nit - framework name seems confusing in relation to the framework name in use from earlier i.e yarn vs local framework. Description in the default-xml for mapreduce.application.framework.path does not mention the need for the URI fragment and how the fragment is used as a sanity check to the classpath. Regarding versions, it seems like users will need to do 2 things. Change the location of the tarball on HDFS and modify the classpath. Users will need to know the exact structure of the classpath. In such a scenario, do defaults even make sense? On the other hand, if we define a common standard i.e. a base path for all MR tarballs, with each tarball in a defined structure ( possibly with version info added on later on for the code to infer the structure of the tarball ), all the user would need to do is specify the base path ( which could have a default value ) and a version which again has a default value. The latter approach would require the code to construct the necessary classpath if the upload path is in use. Do you have any comments on which of the 2 approaches makes more sense? The former is way more flexible but a bit more complex. The latter brittle/inflexible with respect to changing tarball structures but likely more easier to enforce a standard on.
          Hide
          Hitesh Shah added a comment -

          s/Configuration/Jobconf/ in the previous comment.

          Show
          Hitesh Shah added a comment - s/Configuration/Jobconf/ in the previous comment.
          Hide
          Jason Lowe added a comment -

          Thanks for the review, Hitesh!

          Why does classpath need to include all of common, hdfs and yarn jar locations? Assuming that MR is running on a YARN-based cluster, shouldn't the location of the core dependencies come from the cluster deployment i.e. via the env that the NM sets for a container. I believe the only jars that MR should have in its uploaded tarball should be the client jars. I understand that there is no clear boundary for client-side only jars for common and hdfs today ( for For YARN, I believe it should be simple to split out the client-side requirements ) but it is something we should aim for or assume that the jars deployed on the cluster are compatible.

          This is primarily for avoiding jar conflicts and removing dependencies on the nodes. If the cluster upgrades and picks up a new version of jackson/jersey/guava/name-your-favorite-jar-that-breaks-apps-when-updated then that means existing apps can suddenly break due to jar conflicts. Another case we've seen is when a dependency jar is dropped between versions, and apps were depending upon those to be provided by Hadoop. Having the apps provide all of their dependencies means we can focus on just the RPC layer compatibilities (something we have to solve anyway) rather than have to worry as well about the myriad of combinations between jars within the app and those being picked up from the nodes.

          However if desired the user could configure it to work with just a partial tarball by setting the classpath to pickup the jars on the nodes via HADOOP_COMMON_HOME/HADOOP_HDFS_HOME/HADOOP_YARN_HOME references in the classpath like MRApps is doing today.

          I would vote to make the tar-ball in HDFS be the only way to run MR on YARN. Obviously, this cannot be done for 2.x but we should move to this model on trunk and not support the current approach at all there. Comments?

          I'm all for it, and I see this as being a stepping stone to getting there. We'd like to have the ability to run out of HDFS in 2.x as a potential way to do a rolling upgrade of bugfixes in the MR framework. It probably won't be a complete solution to all forms of upgrades (i.e.: what if the client code or ShuffleHandler needs the fix), but it could still be very useful in practice.

          The other point is related to configs.

          Yes, final parameter configs on the nodes conflicting with the job.xml settings are another concern. In practice I don't expect that to be a common issue, but it is something we should try to address in a followup JIRA.

          How do you see framework name extracted from the path to be used? Is it just a safety check to ensure that it is found in the classpath? Will it have any relation to a version?

          I see the framework fragment "alias" primarily used for sanity-checks in case the classpath wasn't updated when using a specified framework and to allow the classpath settings to be a bit more general. For example, ops could configure the classpath once based on an expected framework tarball layout (e.g.: mrframework/share/mapreduce/* : mrframework/share/mapreduce/lib/* etc) and different versions of the tarball can be used without modifying the classpath as long as they match that layout. e.g.: mrtarball-2.3.1.tgz#mrframework, mrtarball-2.3.4.tgz#mrframework, etc. It's sort of like the assumed-layout approach from your last comment. Ops could set the classpath and users could select the framework version without having to set the classpath as long as the layout is compatible. Users could still override the classpath if using a framework that isn't compatible with the assumed layout.

          One problem with the common classpath approach is that the archives need to have the same directory structure, so top-level directories with the version number in them break it. The tarballs deployed to HDFS would have to be reorganized to have a common dir name rather than the versioned name. Not difficult to do, but it is annoying.

          A minor nit - framework name seems confusing in relation to the framework name in use from earlier i.e yarn vs local framework.

          Yeah, that's true. I'm open to suggestions for what to call this instead of framework.

          Regarding versions, it seems like users will need to do 2 things. Change the location of the tarball on HDFS and modify the classpath. Users will need to know the exact structure of the classpath. In such a scenario, do defaults even make sense?

          I wanted this to be flexible so ops/users could decide how to organize the framework (i.e.: partial/complete tarball, monolithic jar, whatever) and be able to set the classpath accordingly. I thought about hardcoding the assumption of the layout, but then that imposes a burden on future frameworks to match that (potentially obsolete) layout and is less flexible. The cost of additional flexibility is additional configuration burden, so it's a tradeoff. That's why I was trying to use the framework fragment alias so the classpath could be set once and hopefully reused for quite some time.

          I suppose one compromise between the two approaches would be a standard file located alongside the archive that the client can read to determine the corresponding classpath settings. There's probably lots of approaches to automatically determining the classpath for an archive, and I'm interested to hear thoughts on it.

          Show
          Jason Lowe added a comment - Thanks for the review, Hitesh! Why does classpath need to include all of common, hdfs and yarn jar locations? Assuming that MR is running on a YARN-based cluster, shouldn't the location of the core dependencies come from the cluster deployment i.e. via the env that the NM sets for a container. I believe the only jars that MR should have in its uploaded tarball should be the client jars. I understand that there is no clear boundary for client-side only jars for common and hdfs today ( for For YARN, I believe it should be simple to split out the client-side requirements ) but it is something we should aim for or assume that the jars deployed on the cluster are compatible. This is primarily for avoiding jar conflicts and removing dependencies on the nodes. If the cluster upgrades and picks up a new version of jackson/jersey/guava/name-your-favorite-jar-that-breaks-apps-when-updated then that means existing apps can suddenly break due to jar conflicts. Another case we've seen is when a dependency jar is dropped between versions, and apps were depending upon those to be provided by Hadoop. Having the apps provide all of their dependencies means we can focus on just the RPC layer compatibilities (something we have to solve anyway) rather than have to worry as well about the myriad of combinations between jars within the app and those being picked up from the nodes. However if desired the user could configure it to work with just a partial tarball by setting the classpath to pickup the jars on the nodes via HADOOP_COMMON_HOME/HADOOP_HDFS_HOME/HADOOP_YARN_HOME references in the classpath like MRApps is doing today. I would vote to make the tar-ball in HDFS be the only way to run MR on YARN. Obviously, this cannot be done for 2.x but we should move to this model on trunk and not support the current approach at all there. Comments? I'm all for it, and I see this as being a stepping stone to getting there. We'd like to have the ability to run out of HDFS in 2.x as a potential way to do a rolling upgrade of bugfixes in the MR framework. It probably won't be a complete solution to all forms of upgrades (i.e.: what if the client code or ShuffleHandler needs the fix), but it could still be very useful in practice. The other point is related to configs. Yes, final parameter configs on the nodes conflicting with the job.xml settings are another concern. In practice I don't expect that to be a common issue, but it is something we should try to address in a followup JIRA. How do you see framework name extracted from the path to be used? Is it just a safety check to ensure that it is found in the classpath? Will it have any relation to a version? I see the framework fragment "alias" primarily used for sanity-checks in case the classpath wasn't updated when using a specified framework and to allow the classpath settings to be a bit more general. For example, ops could configure the classpath once based on an expected framework tarball layout (e.g.: mrframework/share/mapreduce/* : mrframework/share/mapreduce/lib/* etc) and different versions of the tarball can be used without modifying the classpath as long as they match that layout. e.g.: mrtarball-2.3.1.tgz#mrframework, mrtarball-2.3.4.tgz#mrframework, etc. It's sort of like the assumed-layout approach from your last comment. Ops could set the classpath and users could select the framework version without having to set the classpath as long as the layout is compatible. Users could still override the classpath if using a framework that isn't compatible with the assumed layout. One problem with the common classpath approach is that the archives need to have the same directory structure, so top-level directories with the version number in them break it. The tarballs deployed to HDFS would have to be reorganized to have a common dir name rather than the versioned name. Not difficult to do, but it is annoying. A minor nit - framework name seems confusing in relation to the framework name in use from earlier i.e yarn vs local framework. Yeah, that's true. I'm open to suggestions for what to call this instead of framework. Regarding versions, it seems like users will need to do 2 things. Change the location of the tarball on HDFS and modify the classpath. Users will need to know the exact structure of the classpath. In such a scenario, do defaults even make sense? I wanted this to be flexible so ops/users could decide how to organize the framework (i.e.: partial/complete tarball, monolithic jar, whatever) and be able to set the classpath accordingly. I thought about hardcoding the assumption of the layout, but then that imposes a burden on future frameworks to match that (potentially obsolete) layout and is less flexible. The cost of additional flexibility is additional configuration burden, so it's a tradeoff. That's why I was trying to use the framework fragment alias so the classpath could be set once and hopefully reused for quite some time. I suppose one compromise between the two approaches would be a standard file located alongside the archive that the client can read to determine the corresponding classpath settings. There's probably lots of approaches to automatically determining the classpath for an archive, and I'm interested to hear thoughts on it.
          Hide
          Hitesh Shah added a comment -

          Jason Lowe Thanks for the detailed answers to my queries.

          I believe this initial patch is a good start to making MR a user-land library. As it stands, it provides the additional flexibility which can be used by anyone to deploy MR with either the full tarball or a mix-match approach. Though it might be good to have some documentation on the 2 possible approaches ( full tarball vs MR tarball ) and explain how the classpath should be setup.

          Depending on your viewpoint, the classpath-to-hdfs path mapping - whether it comes in from an additional file on HDFS could be considered in a follow-up jira if others believe this is a better solution.

          The one thing to change in the patch is the documentation for mapreduce.application.framework.path - it does not mention the use of the URI fragment and how that interacts with the configured classpath.

          Could you file a follow-up jira for the config handling?

          Show
          Hitesh Shah added a comment - Jason Lowe Thanks for the detailed answers to my queries. I believe this initial patch is a good start to making MR a user-land library. As it stands, it provides the additional flexibility which can be used by anyone to deploy MR with either the full tarball or a mix-match approach. Though it might be good to have some documentation on the 2 possible approaches ( full tarball vs MR tarball ) and explain how the classpath should be setup. Depending on your viewpoint, the classpath-to-hdfs path mapping - whether it comes in from an additional file on HDFS could be considered in a follow-up jira if others believe this is a better solution. The one thing to change in the patch is the documentation for mapreduce.application.framework.path - it does not mention the use of the URI fragment and how that interacts with the configured classpath. Could you file a follow-up jira for the config handling?
          Hide
          Jason Lowe added a comment -

          Thanks for the feedback, Hitesh. I updated the patch with a separate documentation page about the current state of deploying MapReduce via HDFS, limitations of that approach, and how mapreduce.application.framework.path and mapreduce.application.classpath interact. I also updated the description of those configs in mapred-default.xml.

          I filed MAPREDUCE-5534 to track the config handling issue.

          Show
          Jason Lowe added a comment - Thanks for the feedback, Hitesh. I updated the patch with a separate documentation page about the current state of deploying MapReduce via HDFS, limitations of that approach, and how mapreduce.application.framework.path and mapreduce.application.classpath interact. I also updated the description of those configs in mapred-default.xml. I filed MAPREDUCE-5534 to track the config handling issue.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12605111/MAPREDUCE-4421-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4046//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4046//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605111/MAPREDUCE-4421-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4046//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4046//console This message is automatically generated.
          Hide
          Hitesh Shah added a comment -

          Sorry for the delay in the review.

          Regarding addMRFrameworkToDistributedCache() - one minor question: the code allows for a non-qualified URI. Should we enforce provision of a fully-qualified path always?

          Minor nit: I believe there should be nothing in the implementation that requires HDFS as the storage for the MR tarball? Documentation needs to change as a result unless you believe there are reasons for not mentioning other filesystems ( except maybe from a testing point of view )?

          Patch looks good otherwise. Thanks for adding the detailed docs.

          Show
          Hitesh Shah added a comment - Sorry for the delay in the review. Regarding addMRFrameworkToDistributedCache() - one minor question: the code allows for a non-qualified URI. Should we enforce provision of a fully-qualified path always? Minor nit: I believe there should be nothing in the implementation that requires HDFS as the storage for the MR tarball? Documentation needs to change as a result unless you believe there are reasons for not mentioning other filesystems ( except maybe from a testing point of view )? Patch looks good otherwise. Thanks for adding the detailed docs.
          Hide
          Jason Lowe added a comment -

          Thanks for taking another look, Hitesh.

          Regarding addMRFrameworkToDistributedCache() - one minor question: the code allows for a non-qualified URI. Should we enforce provision of a fully-qualified path always?

          I thought it would be easier to let it be qualified by the cluster's configured defaults if not already fully qualified. Otherwise users/admins would have to not only say "hdfs:/path/to/archive" but "hdfs://namenode:port/path/to/archive" and if/when the name or port of the filesystem changes then it breaks. If we let it be qualified by cluster defaults then admins can update the default filesystem in core-site and the simpler forms continue to work unmodified.

          Minor nit: I believe there should be nothing in the implementation that requires HDFS as the storage for the MR tarball?

          Good point. I updated the documentation to refer to a distributed cache deploy rather than an HDFS deploy. However I did call out in the docs the performance ramifications of not using the cluster's default filesystem and a publicly-readable path for the archive. Otherwise the job submitter could end up re-uploading and the nodes re-localizing the framework for each job or each user. It will work, but it will be slower than necessary.

          Show
          Jason Lowe added a comment - Thanks for taking another look, Hitesh. Regarding addMRFrameworkToDistributedCache() - one minor question: the code allows for a non-qualified URI. Should we enforce provision of a fully-qualified path always? I thought it would be easier to let it be qualified by the cluster's configured defaults if not already fully qualified. Otherwise users/admins would have to not only say "hdfs:/path/to/archive" but "hdfs://namenode:port/path/to/archive" and if/when the name or port of the filesystem changes then it breaks. If we let it be qualified by cluster defaults then admins can update the default filesystem in core-site and the simpler forms continue to work unmodified. Minor nit: I believe there should be nothing in the implementation that requires HDFS as the storage for the MR tarball? Good point. I updated the documentation to refer to a distributed cache deploy rather than an HDFS deploy. However I did call out in the docs the performance ramifications of not using the cluster's default filesystem and a publicly-readable path for the archive. Otherwise the job submitter could end up re-uploading and the nodes re-localizing the framework for each job or each user. It will work, but it will be slower than necessary.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12606129/MAPREDUCE-4421-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4071//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4071//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606129/MAPREDUCE-4421-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4071//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4071//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          One minor tweak to the documentation where we need to add $HADOOP_CONF_DIR to the full tarball classpath case otherwise the job will not run on a secure cluster. UserGroupInformation has a static block that loads the security details from a new Configuration, so for now we still need to pick up the security settings from the node's configs.

          Show
          Jason Lowe added a comment - One minor tweak to the documentation where we need to add $HADOOP_CONF_DIR to the full tarball classpath case otherwise the job will not run on a secure cluster. UserGroupInformation has a static block that loads the security details from a new Configuration, so for now we still need to pick up the security settings from the node's configs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12606146/MAPREDUCE-3927-4.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4072//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606146/MAPREDUCE-3927-4.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4072//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Doh, accidentally grabbed the wrong patch.

          Show
          Jason Lowe added a comment - Doh, accidentally grabbed the wrong patch.
          Hide
          Hitesh Shah added a comment -

          Jason Lowe Thanks for the clarification. I believe the performance issues should hold regardless of any filesystem implementation used as long as the distributed cache layer ends up correctly interpreting the permissions to the appropriate LocalResource visibility.

          +1. Latest patch looks good to me.

          Let me know if you are waiting on anyone else to chime in on this. If not, please feel free to go ahead and commit or I shall commit later today.

          Show
          Hitesh Shah added a comment - Jason Lowe Thanks for the clarification. I believe the performance issues should hold regardless of any filesystem implementation used as long as the distributed cache layer ends up correctly interpreting the permissions to the appropriate LocalResource visibility. +1. Latest patch looks good to me. Let me know if you are waiting on anyone else to chime in on this. If not, please feel free to go ahead and commit or I shall commit later today.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12606194/MAPREDUCE-4421-4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4075//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4075//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12606194/MAPREDUCE-4421-4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4075//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4075//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Thanks, Hitesh! I don't know of any others waiting to chime in, so I'll commit this shortly.

          Show
          Jason Lowe added a comment - Thanks, Hitesh! I don't know of any others waiting to chime in, so I'll commit this shortly.
          Hide
          Jason Lowe added a comment -

          Updating the title to describe the change being committed since it does not completely remove the dependency on deployed MR jars. This change still relies on the job client and ShuffleHandler that was deployed to the cluster.

          Show
          Jason Lowe added a comment - Updating the title to describe the change being committed since it does not completely remove the dependency on deployed MR jars. This change still relies on the job client and ShuffleHandler that was deployed to the cluster.
          Hide
          Jason Lowe added a comment -

          I committed this to trunk and branch-2.

          Show
          Jason Lowe added a comment - I committed this to trunk and branch-2.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4506 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4506/)
          MAPREDUCE-4421. Run MapReduce framework via the distributed cache. Contributed by Jason Lowe (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528237)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
          • /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4506 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4506/ ) MAPREDUCE-4421 . Run MapReduce framework via the distributed cache. Contributed by Jason Lowe (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528237 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #350 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/350/)
          MAPREDUCE-4421. Run MapReduce framework via the distributed cache. Contributed by Jason Lowe (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528237)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
          • /hadoop/common/trunk/hadoop-project/src/site/site.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #350 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/350/ ) MAPREDUCE-4421 . Run MapReduce framework via the distributed cache. Contributed by Jason Lowe (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528237 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm /hadoop/common/trunk/hadoop-project/src/site/site.xml

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development