Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
    • Release Note:
      .hadooprc allows users a convenient way to set and/or override the shell level settings.

      Description

      The system should be able to read in user-defined env vars from ~/.hadooprc.

      1. HADOOP-11353.patch
        1 kB
        Allen Wittenauer
      2. HADOOP-11353-01.patch
        2 kB
        Allen Wittenauer
      3. HADOOP-11353-02.patch
        1 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1990 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1990/)
          HADOOP-11353. Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1990 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1990/ ) HADOOP-11353 . Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #40 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/40/)
          HADOOP-11353. Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #40 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/40/ ) HADOOP-11353 . Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #36 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/36/)
          HADOOP-11353. Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #36 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/36/ ) HADOOP-11353 . Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1970 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1970/)
          HADOOP-11353. Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1970 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1970/ ) HADOOP-11353 . Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed) hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #773 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/773/)
          HADOOP-11353. Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #773 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/773/ ) HADOOP-11353 . Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/38/)
          HADOOP-11353. Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/38/ ) HADOOP-11353 . Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6697 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6697/)
          HADOOP-11353. Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6697 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6697/ ) HADOOP-11353 . Add support for .hadooprc (aw) (aw: rev 8e9a2668195d69e30cfce51fda6ad2bd6c1466ed) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          aw Allen Wittenauer added a comment -

          -02:

          • Same as the first version, except now with a --debug statement that says the user's .hadooprc was applied

          I'll be committing this shortly.

          Show
          aw Allen Wittenauer added a comment - -02: Same as the first version, except now with a --debug statement that says the user's .hadooprc was applied I'll be committing this shortly.
          Hide
          aw Allen Wittenauer added a comment -

          Unless anyone has a different opinion, I think I'm going to commit the original version of this patch and let the doc issue be handled by HADOOP-10908, since that's a much more comprehensive doc change.

          Show
          aw Allen Wittenauer added a comment - Unless anyone has a different opinion, I think I'm going to commit the original version of this patch and let the doc issue be handled by HADOOP-10908 , since that's a much more comprehensive doc change.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12685876/HADOOP-11353-01.patch
          against trunk revision ddffcd8.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 65 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5195//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5195//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5195//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685876/HADOOP-11353-01.patch against trunk revision ddffcd8. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 65 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5195//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5195//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5195//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • Add some docs to CommandManual.apt.vm

          I'll be opening another JIRA to update this for other changes.

          Show
          aw Allen Wittenauer added a comment - -01: Add some docs to CommandManual.apt.vm I'll be opening another JIRA to update this for other changes.
          Hide
          aw Allen Wittenauer added a comment -

          I'm expecting more people to use it to replace HADOOP_CLIENT_OPTS than anything else.

          Pointing to a different cluster sort of works; the java portions of HADOOP_CONF_DIR and the paths used for classpath calculations are used after .hadooprc gets processed. So theoretically, you could do something fun like:

          /sw/hadoop.${CLUSTER}
          /etc/hadoop -> dir that only contains a generic hadoop-env.sh
          /etc/hadoop.mycoolcluster
          
          .hadooprc:
          HADOOP_CONF_DIR=/etc/hadoop.${CLUSTER}
          HADOOP_COMMON_HOME=/sw/hadoop.${CLUSTER}
          HADOOP_HDFS_HOME=/sw/hadoop.${CLUSTER}
          HADOOP_YARN_HOME=/sw/hadoop.${CLUSTER}
          HADOOP_MAPRED_HOME=/sw/hadoop.${CLUSTER}
          
          $ CLUSTER=mycoolcluster yarn jar myjob.jar
          

          For documentation, do think it should be in the CommandsManual? I've got a bunch of stuff to document, actually...

          Show
          aw Allen Wittenauer added a comment - I'm expecting more people to use it to replace HADOOP_CLIENT_OPTS than anything else. Pointing to a different cluster sort of works; the java portions of HADOOP_CONF_DIR and the paths used for classpath calculations are used after .hadooprc gets processed. So theoretically, you could do something fun like: /sw/hadoop.${CLUSTER} /etc/hadoop -> dir that only contains a generic hadoop-env.sh /etc/hadoop.mycoolcluster .hadooprc: HADOOP_CONF_DIR=/etc/hadoop.${CLUSTER} HADOOP_COMMON_HOME=/sw/hadoop.${CLUSTER} HADOOP_HDFS_HOME=/sw/hadoop.${CLUSTER} HADOOP_YARN_HOME=/sw/hadoop.${CLUSTER} HADOOP_MAPRED_HOME=/sw/hadoop.${CLUSTER} $ CLUSTER=mycoolcluster yarn jar myjob.jar For documentation, do think it should be in the CommandsManual? I've got a bunch of stuff to document, actually...
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12685164/HADOOP-11353.patch
          against trunk revision 0653918.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5160//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5160//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685164/HADOOP-11353.patch against trunk revision 0653918. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5160//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5160//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          it's an interesting mix of -sh from conf; makes it tricky to edit your env vars to point to a different cluster -but I can see how it'd be hard to pull off.

          given there's no harm in adding the feature, I am provisionally +1 -—though we are going to need some documentation on how the shell works, otherwise this becomes another "secret fact known only if you look at the source" feature

          Show
          stevel@apache.org Steve Loughran added a comment - it's an interesting mix of -sh from conf; makes it tricky to edit your env vars to point to a different cluster -but I can see how it'd be hard to pull off. given there's no harm in adding the feature, I am provisionally +1 -—though we are going to need some documentation on how the shell works, otherwise this becomes another "secret fact known only if you look at the source" feature
          Hide
          aw Allen Wittenauer added a comment -

          -00:

          • Initial version.

          It essentially supports overrides/appends to stuff that would normally be handled in hadoop-env.sh. You can override HADOOP_CONF_DIR, but hadoop-env.sh will be from pre-override.

          Show
          aw Allen Wittenauer added a comment - -00: Initial version. It essentially supports overrides/appends to stuff that would normally be handled in hadoop-env.sh. You can override HADOOP_CONF_DIR, but hadoop-env.sh will be from pre-override.
          Hide
          aw Allen Wittenauer added a comment -

          In order to do this to get complete coverage, the content is needed in two places:

          • prior to hadoop_bootstrap_init (HADOOP_PREFIX and other base vars)
          • after hadoop_exec_userfuncs.

          So this either means reading it twice, actually using two different files, being very very smart about how that file gets read, or just saying "only X vars are supported".

          Show
          aw Allen Wittenauer added a comment - In order to do this to get complete coverage, the content is needed in two places: prior to hadoop_bootstrap_init (HADOOP_PREFIX and other base vars) after hadoop_exec_userfuncs. So this either means reading it twice, actually using two different files, being very very smart about how that file gets read, or just saying "only X vars are supported".

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development