Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Support for shell profiles has been added. They allow for easy integration of additional functionality, classpaths, and more from inside the bash scripts rather than relying upon modifying hadoop-env.sh, etc. See the Unix Shell Guide for more information.
      Show
      Support for shell profiles has been added. They allow for easy integration of additional functionality, classpaths, and more from inside the bash scripts rather than relying upon modifying hadoop-env.sh, etc. See the Unix Shell Guide for more information.

      Description

      It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime.

      1. HADOOP-11485-00.patch
        28 kB
        Allen Wittenauer
      2. HADOOP-11485-01.patch
        28 kB
        Allen Wittenauer
      3. HADOOP-11485-02.patch
        35 kB
        Allen Wittenauer
      4. HADOOP-11485-03.patch
        36 kB
        Allen Wittenauer
      5. HADOOP-11485-04.patch
        36 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment - - edited

          I'm specifically thinking of doing something like this:

          • directory off of HADOOP_CONF_DIR or HADOOP_LIBEXEC_DIR that contains shell fragments (think /etc/profile.d)
          • initializer that uses hadoop_add_colonpath to add a prefix to a var (HADOOP_SHELLFRAG or something)
          • each shell frag could define the following functions:
            _(frag)_hadoop_classpath
            _(frag)_hadoop_init
            _(frag)_hadoop_finalizer
            

          i.e., the current hadoop_add_to_classpath_yarn would get moved out of hadoop-functions.sh into this fragment file and renamed to _yarn_hadoop_classpath.

          A few other notes:

          • HADOOP_CONF_DIR would need to get moved to first in to last in+prepend. It must always be first in the classpath. We don't want 3rd parties coming in front.
          • We could provide no guarantees, really, as to when a jar appears in the classpath using this method. So this wouldn't be a way to override classes.
          • Currently, the only way to manage $@ via this fragment is going to be on source'ing it. So sourcing will happen after we do our normal shell param processing. So hadoop-common options will need to come first, 3rd party after, followed by the appropriate shell subcommand. e.g., yarn --conf foo --hbaseconf bar jar myhbase.jar
          • We'll likely need to banish all of the 'extraneous' shell env vars introduced in 2.x and effectively deprecated as part of HADOOP-9902.
          Show
          aw Allen Wittenauer added a comment - - edited I'm specifically thinking of doing something like this: directory off of HADOOP_CONF_DIR or HADOOP_LIBEXEC_DIR that contains shell fragments (think /etc/profile.d) initializer that uses hadoop_add_colonpath to add a prefix to a var (HADOOP_SHELLFRAG or something) each shell frag could define the following functions: _(frag)_hadoop_classpath _(frag)_hadoop_init _(frag)_hadoop_finalizer i.e., the current hadoop_add_to_classpath_yarn would get moved out of hadoop-functions.sh into this fragment file and renamed to _yarn_hadoop_classpath. A few other notes: HADOOP_CONF_DIR would need to get moved to first in to last in+prepend. It must always be first in the classpath. We don't want 3rd parties coming in front. We could provide no guarantees, really, as to when a jar appears in the classpath using this method. So this wouldn't be a way to override classes. Currently, the only way to manage $@ via this fragment is going to be on source'ing it. So sourcing will happen after we do our normal shell param processing. So hadoop-common options will need to come first, 3rd party after, followed by the appropriate shell subcommand. e.g., yarn --conf foo --hbaseconf bar jar myhbase.jar We'll likely need to banish all of the 'extraneous' shell env vars introduced in 2.x and effectively deprecated as part of HADOOP-9902 .
          Hide
          aw Allen Wittenauer added a comment -

          -00:

          • First pass at this functionality to get a sense if this is the right approach
          • yarn, hdfs, and mapred classpath and yarn finalizer moved to frag files
          • KNOWN BUG: hdfs-fragment doesn't get moved into libexec/shellfrags as part of the distribution (grumble grumble maven grumble grumble)
          • KNOWN BUG: missing license info at the top
          • Stripped ending line spaces

          A few notes:

          • YARN properties are now instantiated on every java command line.
          • This doesn't deprecate anything, as that should be handled in that other jira.
          Show
          aw Allen Wittenauer added a comment - -00: First pass at this functionality to get a sense if this is the right approach yarn, hdfs, and mapred classpath and yarn finalizer moved to frag files KNOWN BUG: hdfs-fragment doesn't get moved into libexec/shellfrags as part of the distribution (grumble grumble maven grumble grumble) KNOWN BUG: missing license info at the top Stripped ending line spaces A few notes: YARN properties are now instantiated on every java command line. This doesn't deprecate anything, as that should be handled in that other jira.
          Hide
          aw Allen Wittenauer added a comment -

          I'm making HADOOP-11460 required, as it is nearly impossible to give 3rd parties guidance on what they can/cannot change without removing some of the cruft generated by previous shell work.

          Show
          aw Allen Wittenauer added a comment - I'm making HADOOP-11460 required, as it is nearly impossible to give 3rd parties guidance on what they can/cannot change without removing some of the cruft generated by previous shell work.
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • Renamed shell frags to shell profiles
          • added new shellprofile.d dirs to both source and distribution trees (so hdfs should work now)
          • renamed hadoop_boostrap_init to just hadoop_bootstrap to help clarify its purpose and remove potential confusion with other init functions
          • added support for HADOOP_CONF_DIR/shellprofile.d. should make it easier for 3rd parties to inject themselves rather than deep in the bowels of hadoop
          Show
          aw Allen Wittenauer added a comment - -01: Renamed shell frags to shell profiles added new shellprofile.d dirs to both source and distribution trees (so hdfs should work now) renamed hadoop_boostrap_init to just hadoop_bootstrap to help clarify its purpose and remove potential confusion with other init functions added support for HADOOP_CONF_DIR/shellprofile.d. should make it easier for 3rd parties to inject themselves rather than deep in the bowels of hadoop
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693179/HADOOP-11485-01.patch
          against trunk revision cb0a15d.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ha.TestZKFailoverController

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//artifact/patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693179/HADOOP-11485-01.patch against trunk revision cb0a15d. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5432//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -02:

          • Add native lib support
          • provide an example sitting in HADOOP_CONF_DIR
          • fix the missing licenses (woops)
          • add some debug statements
          Show
          aw Allen Wittenauer added a comment - -02: Add native lib support provide an example sitting in HADOOP_CONF_DIR fix the missing licenses (woops) add some debug statements
          Hide
          aw Allen Wittenauer added a comment -

          BTW, I'm debating merging this patch and HADOOP-11460 into one patch.

          Show
          aw Allen Wittenauer added a comment - BTW, I'm debating merging this patch and HADOOP-11460 into one patch.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Sorry, this is probably a dumb question, but what "third parties" do we want to be plugging shell commands in to ./bin/hadoop ? Can't these guys just use "yarn classpath" and / or "hadoop classpath" to get the Hadoop classpath, and run their own jars? I'm not sure I see a benefit to the increased complexity... maybe I'm missing something.

          Show
          cmccabe Colin P. McCabe added a comment - Sorry, this is probably a dumb question, but what "third parties" do we want to be plugging shell commands in to ./bin/hadoop ? Can't these guys just use "yarn classpath" and / or "hadoop classpath" to get the Hadoop classpath, and run their own jars? I'm not sure I see a benefit to the increased complexity... maybe I'm missing something.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693327/HADOOP-11485-02.patch
          against trunk revision c94c0d2.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ha.TestZKFailoverController
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5435//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5435//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693327/HADOOP-11485-02.patch against trunk revision c94c0d2. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5435//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5435//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          Let's say I'm an enterprise user that wants to use three other commercial products with my commercial Hadoop distribution: an input format, an encryption library, and an auth handler. Today, doing this requires the Hadoop admin to modify HADOOP_USER_CLASSPATH or shove the contents into the Hadoop binary distribution directories in order to get the daemons to always have these jars available.

          HADOOP_USER_CLASSPATH is extra bad because this means that the end users of the system will also need to take care that they don't drop these extra classpaths if they also need custom jars for their job.

          By making this pluggable, 3rd parties can inject their bits in a MUCH safer way plus the end user controls are left open for them to do whatever with.

          Show
          aw Allen Wittenauer added a comment - Let's say I'm an enterprise user that wants to use three other commercial products with my commercial Hadoop distribution: an input format, an encryption library, and an auth handler. Today, doing this requires the Hadoop admin to modify HADOOP_USER_CLASSPATH or shove the contents into the Hadoop binary distribution directories in order to get the daemons to always have these jars available. HADOOP_USER_CLASSPATH is extra bad because this means that the end users of the system will also need to take care that they don't drop these extra classpaths if they also need custom jars for their job. By making this pluggable, 3rd parties can inject their bits in a MUCH safer way plus the end user controls are left open for them to do whatever with.
          Hide
          pocky John Smith added a comment -

          +1 (non-binding)

          Show
          pocky John Smith added a comment - +1 (non-binding)
          Hide
          aw Allen Wittenauer added a comment -

          Given no further comments, I'll commit this next week.

          Show
          aw Allen Wittenauer added a comment - Given no further comments, I'll commit this next week.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi, Allen Wittenauer. Thank you for doing this. This looks great.

          I took it for a test run with a concrete use case that I mentioned on another jira: adding hadoop-azure to the classpath at distro time. Everything worked as advertised in the example. I read through that and then dropped the following file at shellprofile.d/hadoop_azure:

          hadoop_add_profile hadoop_azure
          
          function _hadoop_azure_hadoop_classpath
          {
            hadoop_add_classpath "${HADOOP_PREFIX}/share/hadoop/tools/lib/hadoop-azure-3.0.0-SNAPSHOT.jar" after
            hadoop_add_classpath "${HADOOP_PREFIX}/share/hadoop/tools/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar" after
          }
          

          With that in place, I can run fs shell commands that access paths in Azure Storage. This is easier than the existing options for trying to achieve this kind of customization.

          I have only 2 meta-comments, which don't necessarily need to be addressed in the scope of this jira:

          1. With the introduction of shell profiles, we essentially have a new public API contract. We'll need to maintain compatibility so that we don't break existing shell profiles during routine code maintenance. (i.e. hadoop_add_classpath can't be renamed within a major version). Maybe we need a commenting convention, similar to the interface visibility and stability annotations that we use in Java, to help guide developers.
          2. I think we'll also eventually need user-facing documentation on hadoop.apache.org to describe the new functionality in the shell rewrite. For shell profile implementors, something like API docs would be helpful to let people know about useful things like hadoop_debug.

          I think the current patch needs a small rebase. After that, +1.

          Show
          cnauroth Chris Nauroth added a comment - Hi, Allen Wittenauer . Thank you for doing this. This looks great. I took it for a test run with a concrete use case that I mentioned on another jira: adding hadoop-azure to the classpath at distro time. Everything worked as advertised in the example. I read through that and then dropped the following file at shellprofile.d/hadoop_azure: hadoop_add_profile hadoop_azure function _hadoop_azure_hadoop_classpath { hadoop_add_classpath "${HADOOP_PREFIX}/share/hadoop/tools/lib/hadoop-azure-3.0.0-SNAPSHOT.jar" after hadoop_add_classpath "${HADOOP_PREFIX}/share/hadoop/tools/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar" after } With that in place, I can run fs shell commands that access paths in Azure Storage. This is easier than the existing options for trying to achieve this kind of customization. I have only 2 meta-comments, which don't necessarily need to be addressed in the scope of this jira: With the introduction of shell profiles, we essentially have a new public API contract. We'll need to maintain compatibility so that we don't break existing shell profiles during routine code maintenance. (i.e. hadoop_add_classpath can't be renamed within a major version). Maybe we need a commenting convention, similar to the interface visibility and stability annotations that we use in Java, to help guide developers. I think we'll also eventually need user-facing documentation on hadoop.apache.org to describe the new functionality in the shell rewrite. For shell profile implementors, something like API docs would be helpful to let people know about useful things like hadoop_debug . I think the current patch needs a small rebase. After that, +1.
          Hide
          aw Allen Wittenauer added a comment -

          I did see a couple minor (non-interface) things in the example file, which I'll fix.

          Thanks for the review. I'll fix those minor issues and rebase the patch.

          On your two points, yeah, with this we're starting to approach a point where need to probably formalize the shell code now. I've opened HADOOP-11553 to start working on that.

          Show
          aw Allen Wittenauer added a comment - I did see a couple minor (non-interface) things in the example file, which I'll fix. Thanks for the review. I'll fix those minor issues and rebase the patch. On your two points, yeah, with this we're starting to approach a point where need to probably formalize the shell code now. I've opened HADOOP-11553 to start working on that.
          Hide
          aw Allen Wittenauer added a comment -

          -03:

          • Rebase
          • Fix some minor problems with the example
          • Remove some deprecated vars now that HADOOP-11460 is committed
          • Fix some shellcheck errors (woops)
          Show
          aw Allen Wittenauer added a comment - -03: Rebase Fix some minor problems with the example Remove some deprecated vars now that HADOOP-11460 is committed Fix some shellcheck errors (woops)
          Hide
          cnauroth Chris Nauroth added a comment -

          I think v03 has some issues with usage of hadoop_translate_cygwin_path in shellprofile.d/yarn:

            hadoop_translate_cygwin_path "${yld}"
          

          Inside hadoop_translate_cygwin_path, the argument is part of an eval for setting (possibly changing) the variable value, so instead of expanding the variable's value at the call site, we just need to pass through the variable name:

            hadoop_translate_cygwin_path yld
          

          Also, I think I see a typo, defining variable yhd, but then trying to use ygd:

            local yhd=$HADOOP_YARN_HOME
            hadoop_translate_cygwin_path "${ygd}"
          
          Show
          cnauroth Chris Nauroth added a comment - I think v03 has some issues with usage of hadoop_translate_cygwin_path in shellprofile.d/yarn: hadoop_translate_cygwin_path "${yld}" Inside hadoop_translate_cygwin_path , the argument is part of an eval for setting (possibly changing) the variable value, so instead of expanding the variable's value at the call site, we just need to pass through the variable name: hadoop_translate_cygwin_path yld Also, I think I see a typo, defining variable yhd , but then trying to use ygd : local yhd=$HADOOP_YARN_HOME hadoop_translate_cygwin_path "${ygd}"
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696956/HADOOP-11485-03.patch
          against trunk revision af3aadf.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5612//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5612//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696956/HADOOP-11485-03.patch against trunk revision af3aadf. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5612//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5612//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -04:

          • ugh, I really did screw up yarn as part of fixing the deprecated vars. also, shame on me for forgetting that the cygwin path function takes a reference. hah. should be fixed now.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - -04: ugh, I really did screw up yarn as part of fixing the deprecated vars. also, shame on me for forgetting that the cygwin path function takes a reference. hah. should be fixed now. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697103/HADOOP-11485-04.patch
          against trunk revision c1957fe.

          -1 @author. The patch appears to contain @author tags which the Hadoop community has agreed to not allow in code contributions.

          +1 tests included. The patch appears to include new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5616//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697103/HADOOP-11485-04.patch against trunk revision c1957fe. -1 @author . The patch appears to contain @author tags which the Hadoop community has agreed to not allow in code contributions. +1 tests included . The patch appears to include new or modified test files. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5616//console This message is automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch -04. I retested Cygwin on Windows, and all is well. I'm not sure what's up with that last Jenkins run. The patch definitely applies clean on current trunk for me.

          Thanks again, Allen.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch -04. I retested Cygwin on Windows, and all is well. I'm not sure what's up with that last Jenkins run. The patch definitely applies clean on current trunk for me. Thanks again, Allen.
          Hide
          aw Allen Wittenauer added a comment -

          Thanks for the review!

          Committed to trunk.

          Show
          aw Allen Wittenauer added a comment - Thanks for the review! Committed to trunk.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7044 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7044/)
          HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-mapreduce-project/shellprofile.d/mapreduce
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7044 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7044/ ) HADOOP-11485 . Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs hadoop-common-project/hadoop-common/CHANGES.txt hadoop-mapreduce-project/shellprofile.d/mapreduce hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/97/)
          HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0)

          • hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-mapreduce-project/shellprofile.d/mapreduce
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/97/ ) HADOOP-11485 . Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example hadoop-common-project/hadoop-common/CHANGES.txt hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-mapreduce-project/shellprofile.d/mapreduce hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #831 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/831/)
          HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example
          • hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • hadoop-mapreduce-project/shellprofile.d/mapreduce
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #831 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/831/ ) HADOOP-11485 . Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml hadoop-mapreduce-project/shellprofile.d/mapreduce hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2029 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2029/)
          HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-mapreduce-project/shellprofile.d/mapreduce
          • hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          • hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2029 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2029/ ) HADOOP-11485 . Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-common-project/hadoop-common/CHANGES.txt hadoop-mapreduce-project/shellprofile.d/mapreduce hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #94 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/94/)
          HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • hadoop-mapreduce-project/shellprofile.d/mapreduce
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example
          • hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #94 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/94/ ) HADOOP-11485 . Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml hadoop-mapreduce-project/shellprofile.d/mapreduce hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/98/)
          HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0)

          • hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn
          • hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-mapreduce-project/shellprofile.d/mapreduce
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/98/ ) HADOOP-11485 . Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-mapreduce-project/shellprofile.d/mapreduce hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2048 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2048/)
          HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn
          • hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example
          • hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-mapreduce-project/shellprofile.d/mapreduce
          • hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2048 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2048/ ) HADOOP-11485 . Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-mapreduce-project/shellprofile.d/mapreduce hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #832 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/832/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #832 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/832/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/98/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/98/ )
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2030 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2030/)

          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2030 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2030/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #95 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/95/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #95 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/95/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2049 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2049/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2049 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2049/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/99/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/99/ )

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development