Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9642

Configuration to resolve environment variables via ${env.VARIABLE} references

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta, 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: conf, scripts
    • Labels:
      None

      Description

      We should be able to get env variables from Configuration files, rather than just system properties. I propose using the traditional env prefix {{$

      {env.PATH}

      }} to make it immediately clear to people reading a conf file that it's an env variable -and to avoid any confusion with system properties and existing configuration properties.

      1. HADOOP-9642.001.patch
        5 kB
        Kengo Seki
      2. HADOOP-9642.002.patch
        10 kB
        Kengo Seki

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2113 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2113/)
          HADOOP-9642. Configuration to resolve environment variables via $

          {env.VARIABLE}

          references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2113 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2113/ ) HADOOP-9642 . Configuration to resolve environment variables via $ {env.VARIABLE} references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #164 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/164/)
          HADOOP-9642. Configuration to resolve environment variables via $

          {env.VARIABLE}

          references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #164 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/164/ ) HADOOP-9642 . Configuration to resolve environment variables via $ {env.VARIABLE} references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #154 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/154/)
          HADOOP-9642. Configuration to resolve environment variables via $

          {env.VARIABLE}

          references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #154 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/154/ ) HADOOP-9642 . Configuration to resolve environment variables via $ {env.VARIABLE} references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #897 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/897/)
          HADOOP-9642. Configuration to resolve environment variables via $

          {env.VARIABLE}

          references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #897 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/897/ ) HADOOP-9642 . Configuration to resolve environment variables via $ {env.VARIABLE} references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2095/)
          HADOOP-9642. Configuration to resolve environment variables via $

          {env.VARIABLE}

          references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2095/ ) HADOOP-9642 . Configuration to resolve environment variables via $ {env.VARIABLE} references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/163/)
          HADOOP-9642. Configuration to resolve environment variables via $

          {env.VARIABLE}

          references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/163/ ) HADOOP-9642 . Configuration to resolve environment variables via $ {env.VARIABLE} references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7581 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7581/)
          HADOOP-9642. Configuration to resolve environment variables via $

          {env.VARIABLE}

          references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7581 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7581/ ) HADOOP-9642 . Configuration to resolve environment variables via $ {env.VARIABLE} references (Kengo Seki via aw) (aw: rev d4e8c09593b8edc4d0494d64ce19ac78c6f31277) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          Hide
          aw Allen Wittenauer added a comment -

          +1 lgtm. committed to trunk.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - +1 lgtm. committed to trunk. Thanks!
          Hide
          aw Allen Wittenauer added a comment -

          Linking HADOOP-11553. If both get committed, the docs should reference this functionality.

          Show
          aw Allen Wittenauer added a comment - Linking HADOOP-11553 . If both get committed, the docs should reference this functionality.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706745/HADOOP-9642.002.patch
          against trunk revision 972f1f1.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5986//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5986//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706745/HADOOP-9642.002.patch against trunk revision 972f1f1. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5986//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5986//console This message is automatically generated.
          Hide
          sekikn Kengo Seki added a comment -

          ":-" and "-" behave in the same way as shell's variable expansion.

          Show
          sekikn Kengo Seki added a comment - ":-" and "-" behave in the same way as shell's variable expansion.
          Hide
          sekikn Kengo Seki added a comment -

          Attaching a revised patch.

          • Default value feature added. In addition to ":-", "-" is also supported.
          • Javadoc modified.

          Could anyone review it? Thank you.

          Show
          sekikn Kengo Seki added a comment - Attaching a revised patch. Default value feature added. In addition to ":-", "-" is also supported. Javadoc modified. Could anyone review it? Thank you.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706394/HADOOP-9642.001.patch
          against trunk revision 4cd54d9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5983//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5983//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706394/HADOOP-9642.001.patch against trunk revision 4cd54d9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5983//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5983//console This message is automatically generated.
          Hide
          sekikn Kengo Seki added a comment -

          Attaching a preliminary patch. Default values are not supported yet.

          Show
          sekikn Kengo Seki added a comment - Attaching a preliminary patch. Default values are not supported yet.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm thinking of the specific use case of resolving values passed down via YARN. I don't see why we'd need default values, given they aren't there for system property resolution either.

          I agree it's nondeterministic -but again, we have that problem with system properties

          Show
          stevel@apache.org Steve Loughran added a comment - I'm thinking of the specific use case of resolving values passed down via YARN. I don't see why we'd need default values, given they aren't there for system property resolution either. I agree it's nondeterministic -but again, we have that problem with system properties
          Hide
          daryn Daryn Sharp added a comment -

          Generally sounds reasonable to me. Although you'd probably want additional syntax such as ${ENV:default-value} if the env isn't defined.

          One concern is introduction of non-deterministic behavior. An example being the "works for me" scenario where user A and user B are puzzled why each gets different results from the same command because one of them forgot to set additional envs beyond HADOOP_HOME.

          Along those lines, it does introduce additional logistics if server process configs reference envs. It becomes easier to make mistakes if there's a mix of interactive/non-interactive shells commands to start processes and the most appropriate rc (profile/bashrc/etc) isn't updated. Although it could be argued the admin brought the pain upon himself...

          Show
          daryn Daryn Sharp added a comment - Generally sounds reasonable to me. Although you'd probably want additional syntax such as ${ENV:default-value} if the env isn't defined. One concern is introduction of non-deterministic behavior. An example being the "works for me" scenario where user A and user B are puzzled why each gets different results from the same command because one of them forgot to set additional envs beyond HADOOP_HOME. Along those lines, it does introduce additional logistics if server process configs reference envs. It becomes easier to make mistakes if there's a mix of interactive/non-interactive shells commands to start processes and the most appropriate rc (profile/bashrc/etc) isn't updated. Although it could be argued the admin brought the pain upon himself...

            People

            • Assignee:
              sekikn Kengo Seki
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development