Hadoop Common
  1. Hadoop Common
  2. HADOOP-7711

hadoop-env.sh generated from templates has duplicate info

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang)
    1. HADOOP-7711.20s.patch
      3 kB
      Arpit Gupta
    2. HADOOP-7711.patch
      3 kB
      Arpit Gupta

      Activity

      Hide
      Arpit Gupta added a comment -

      Here is the info, notice how the same info is repeated for the namenode and secondary namenode opts

      1. Command specific options appended to HADOOP_OPTS when specified
        export HADOOP_NAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT "
        HADOOP_JOBTRACKER_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dmapred.audit.logger=INFO,MRAUDIT -

      export HADOOP_SECONDARYNAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT "

      Show
      Arpit Gupta added a comment - Here is the info, notice how the same info is repeated for the namenode and secondary namenode opts Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT " HADOOP_JOBTRACKER_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dmapred.audit.logger=INFO,MRAUDIT - export HADOOP_SECONDARYNAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT "
      Hide
      Eric Yang added a comment -

      The trigger of this problem was fixed in MAPREDUCE-3112. Arpit, could you confirm it?

      Show
      Eric Yang added a comment - The trigger of this problem was fixed in MAPREDUCE-3112 . Arpit, could you confirm it?
      Hide
      Arpit Gupta added a comment -

      Eric this seems to be a different problem.

      hadoop-env.sh template has the following...

      export HADOOP_NAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $

      {HADOOP_NAMENODE_OPTS}

      "

      export HADOOP_SECONDARYNAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $

      {HADOOP_SECONDARYNAMENODE_OPTS}

      "

      Somehow at the time of generating the conf from the template they get all duplicate info.

      Show
      Arpit Gupta added a comment - Eric this seems to be a different problem. hadoop-env.sh template has the following... export HADOOP_NAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $ {HADOOP_NAMENODE_OPTS} " export HADOOP_SECONDARYNAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $ {HADOOP_SECONDARYNAMENODE_OPTS} " Somehow at the time of generating the conf from the template they get all duplicate info.
      Hide
      Arpit Gupta added a comment -

      The info in the first comment was what was in the hadoop-env.sh if they should not be replaced my we change them from

      export HADOOP_NAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $

      {HADOOP_NAMENODE_OPTS}

      "

      to

      export HADOOP_NAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $HADOOP_NAMENODE_OPTS"

      Do you think that might be the cause?

      Show
      Arpit Gupta added a comment - The info in the first comment was what was in the hadoop-env.sh if they should not be replaced my we change them from export HADOOP_NAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $ {HADOOP_NAMENODE_OPTS} " to export HADOOP_NAMENODE_OPTS="-Dsecurity.audit.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $HADOOP_NAMENODE_OPTS" Do you think that might be the cause?
      Hide
      Eric Yang added a comment -

      If the running user environment contains HADOOP_NAMENODE_OPTS, then it will trigger this problem. $HADOOP_NAMENODE_OPTS will eliminate the current problem, but it also enforce $HADOOP_NAMENODE_OPTS to be non-overwritable by the template system.

      Show
      Eric Yang added a comment - If the running user environment contains HADOOP_NAMENODE_OPTS, then it will trigger this problem. $HADOOP_NAMENODE_OPTS will eliminate the current problem, but it also enforce $HADOOP_NAMENODE_OPTS to be non-overwritable by the template system.
      Hide
      Arpit Gupta added a comment -

      Will perform the following.

      1. remove curly parenthesis from OPTS vars in the hadoop-env.sh
      2. unset the those env in the hadoop-setup-conf.sh

      Show
      Arpit Gupta added a comment - Will perform the following. 1. remove curly parenthesis from OPTS vars in the hadoop-env.sh 2. unset the those env in the hadoop-setup-conf.sh
      Hide
      Arpit Gupta added a comment -

      patch for 20 security branch

      Show
      Arpit Gupta added a comment - patch for 20 security branch
      Hide
      Arpit Gupta added a comment -

      patch for trunk. also updated HADOOP_OPTS to use HADOOP_CLIENT_OPTS to make it behave the same as 20 security

      Show
      Arpit Gupta added a comment - patch for trunk. also updated HADOOP_OPTS to use HADOOP_CLIENT_OPTS to make it behave the same as 20 security
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12497482/HADOOP-7711.patch
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in .

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/255//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/255//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497482/HADOOP-7711.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/255//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/255//console This message is automatically generated.
      Hide
      Arpit Gupta added a comment -

      no tests included as this is part of the deployment scripts.

      Show
      Arpit Gupta added a comment - no tests included as this is part of the deployment scripts.
      Hide
      Eric Yang added a comment -

      +1 looks good.

      Show
      Eric Yang added a comment - +1 looks good.
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-trunk-Commit #1010 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1010/)
      HADOOP-7711. Fixed recursive sourcing of HADOOP_OPTS environment
      variables (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Show
      Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1010 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1010/ ) HADOOP-7711 . Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Hide
      Eric Yang added a comment -

      I just committed this to trunk, 0.23 branch, 0.20 security branch, 0.20.205. Thanks Arpit.

      Show
      Eric Yang added a comment - I just committed this to trunk, 0.23 branch, 0.20 security branch, 0.20.205. Thanks Arpit.
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk-Commit #1088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1088/)
      HADOOP-7711. Fixed recursive sourcing of HADOOP_OPTS environment
      variables (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1088/ ) HADOOP-7711 . Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk-Commit #1030 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1030/)
      HADOOP-7711. Fixed recursive sourcing of HADOOP_OPTS environment
      variables (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1030 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1030/ ) HADOOP-7711 . Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk #821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/821/)
      HADOOP-7711. Fixed recursive sourcing of HADOOP_OPTS environment
      variables (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/821/ ) HADOOP-7711 . Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-0.23-Build #30 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/30/)
      HADOOP-7711. Fixed recursive sourcing of HADOOP_OPTS environment
      variables (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178852
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #30 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/30/ ) HADOOP-7711 . Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178852 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-0.23-Build #37 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/)
      HADOOP-7711. Fixed recursive sourcing of HADOOP_OPTS environment
      variables (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178852
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #37 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/ ) HADOOP-7711 . Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178852 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk #851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/)
      HADOOP-7711. Fixed recursive sourcing of HADOOP_OPTS environment
      variables (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/ ) HADOOP-7711 . Fixed recursive sourcing of HADOOP_OPTS environment variables (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178853 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
      Hide
      Matt Foley added a comment -

      Corrected Fix Versions field, based on commit dates.

      Show
      Matt Foley added a comment - Corrected Fix Versions field, based on commit dates.

        People

        • Assignee:
          Arpit Gupta
          Reporter:
          Arpit Gupta
        • Votes:
          0 Vote for this issue
          Watchers:
          0 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development