Hadoop Common
  1. Hadoop Common
  2. HADOOP-8433

Don't set HADOOP_LOG_DIR in hadoop-env.sh

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It's better to comment the following in hadoop-env.sh

      1. Where log files are stored. $HADOOP_HOME/logs by default.
        export HADOOP_LOG_DIR=$ {HADOOP_LOG_DIR}

        /$USER

      Because of this logs are placing under root($user) and this getting called two times while starting process.
      hence logs are placing at /root/root/

      1. Hadoop-8433.patch
        0.7 kB
        Brahma Reddy Battula
      2. HADOOP-8433.patch
        1 kB
        Brahma Reddy Battula

        Activity

        Brahma Reddy Battula created issue -
        Hide
        Brahma Reddy Battula added a comment -

        Solution can be removing $user or commenting HADOOP_LOG_DIR in hadoop-env.sh..

        Show
        Brahma Reddy Battula added a comment - Solution can be removing $user or commenting HADOOP_LOG_DIR in hadoop-env.sh..
        Hide
        Brahma Reddy Battula added a comment -

        Attaching patch by commeting HADOOP_LOG_DIR in haddop-env.sh.

        Show
        Brahma Reddy Battula added a comment - Attaching patch by commeting HADOOP_LOG_DIR in haddop-env.sh.
        Brahma Reddy Battula made changes -
        Field Original Value New Value
        Attachment Hadoop-8433.patch [ 12529565 ]
        Hide
        Brahma Reddy Battula added a comment -

        Attached patch

        Show
        Brahma Reddy Battula added a comment - Attached patch
        Brahma Reddy Battula made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12529565/Hadoop-8433.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1031//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1031//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529565/Hadoop-8433.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1031//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1031//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        Hi Brahma,

        You're seeing /root/root from a tarball install? How did you test this change?

        Thanks,
        ELi

        Show
        Eli Collins added a comment - Hi Brahma, You're seeing /root/root from a tarball install? How did you test this change? Thanks, ELi
        Eli Collins made changes -
        Assignee Brahma Reddy Battula [ brahmareddy ]
        Hide
        Brahma Reddy Battula added a comment -

        I have commented (In haddop-env.sh) since LOG_DIR will be set in hadoop-daemon.sh..Like following..

        if [ ! -w "$HADOOP_LOG_DIR" ] ; then
        mkdir -p "$HADOOP_LOG_DIR"
        chown $HADOOP_IDENT_STRING $HADOOP_LOG_DIR
        fi

        Show
        Brahma Reddy Battula added a comment - I have commented (In haddop-env.sh) since LOG_DIR will be set in hadoop-daemon.sh..Like following.. if [ ! -w "$HADOOP_LOG_DIR" ] ; then mkdir -p "$HADOOP_LOG_DIR" chown $HADOOP_IDENT_STRING $HADOOP_LOG_DIR fi
        Hide
        Brahma Reddy Battula added a comment -

        Hi Eli,

        thanks for look.

        Yes.. Logs are placed /root/root/

        Show
        Brahma Reddy Battula added a comment - Hi Eli, thanks for look. Yes.. Logs are placed /root/root/
        Hide
        Eli Collins added a comment -

        Thanks Brahma, this is building the tarballs using user root or using the packages?

        There are two copies of hadoop-env.sh, let's update both to be consistent.

        Show
        Eli Collins added a comment - Thanks Brahma, this is building the tarballs using user root or using the packages? There are two copies of hadoop-env.sh, let's update both to be consistent.
        Hide
        Brahma Reddy Battula added a comment -

        HI Eli,

        I have updated both places..thanks a lot

        Show
        Brahma Reddy Battula added a comment - HI Eli, I have updated both places..thanks a lot
        Brahma Reddy Battula made changes -
        Attachment HADOOP-8433.patch [ 12529950 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12529950/HADOOP-8433.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1041//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1041//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529950/HADOOP-8433.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1041//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1041//console This message is automatically generated.
        Hide
        Jay Finger added a comment -

        I don't like the approach of commenting this line out. I think a lot of admins will infer from this that uncommenting the line will result in some useful behavior.

        A bigger problem is that hadoop-env.sh is sometimes included twice. For example, hadoop-daemon.sh includes hadoop-config.sh, which includes hadoop-env.sh. Then later hadoop-daemon.sh includes hadoop-env.sh explicitly. It seems to me that this is the actual root cause of the issue, and that it is likely going to cause other headaches in the future by people who don't suspect that it will be included twice.

        Show
        Jay Finger added a comment - I don't like the approach of commenting this line out. I think a lot of admins will infer from this that uncommenting the line will result in some useful behavior. A bigger problem is that hadoop-env.sh is sometimes included twice. For example, hadoop-daemon.sh includes hadoop-config.sh, which includes hadoop-env.sh. Then later hadoop-daemon.sh includes hadoop-env.sh explicitly. It seems to me that this is the actual root cause of the issue, and that it is likely going to cause other headaches in the future by people who don't suspect that it will be included twice.
        Hide
        Brahma Reddy Battula added a comment -

        HI Jay Finger,

        Thanks For look..

        fyi ..I had proposed following two solutions when I raised..

        Solution can be removing $user or commenting HADOOP_LOG_DIR in hadoop-env.sh..

        I am agreeing with you..

        I don't like the approach of commenting this line out. I think a lot of admins will infer from this that uncommenting the line will result in some useful behavior

        It wn't harm by commenting this property since it'll be set in hadoop-daemon.sh...I think it's better to remove $USER..Please let me know..

        A bigger problem is that hadoop-env.sh is sometimes included twice

        Anyway this will be addressed by HADOOP-8464.

        Show
        Brahma Reddy Battula added a comment - HI Jay Finger, Thanks For look.. fyi ..I had proposed following two solutions when I raised.. Solution can be removing $user or commenting HADOOP_LOG_DIR in hadoop-env.sh.. I am agreeing with you.. I don't like the approach of commenting this line out. I think a lot of admins will infer from this that uncommenting the line will result in some useful behavior It wn't harm by commenting this property since it'll be set in hadoop-daemon.sh...I think it's better to remove $USER..Please let me know.. A bigger problem is that hadoop-env.sh is sometimes included twice Anyway this will be addressed by HADOOP-8464 .
        Hide
        Eli Collins added a comment -

        +1 latest patch looks good, I tested it against a tarball install.

        Show
        Eli Collins added a comment - +1 latest patch looks good, I tested it against a tarball install.
        Eli Collins made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Summary Logs are getting misplaced after introducing hadoop-env.sh Don't set HADOOP_LOG_DIR in hadoop-env.sh
        Affects Version/s 3.0.0 [ 12320357 ]
        Target Version/s 2.0.0-alpha, 3.0.0 [ 12320352, 12320357 ] 2.0.1-alpha [ 12321441 ]
        Hide
        Eli Collins added a comment -

        I've committed this and merged to branch-2. Thanks Brahma.

        Show
        Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Brahma.
        Eli Collins made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Target Version/s 2.0.1-alpha [ 12321441 ]
        Fix Version/s 2.0.1-alpha [ 12321441 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2357 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2357/)
        HADOOP-8433. Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2357 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2357/ ) HADOOP-8433 . Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2430 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2430/)
        HADOOP-8433. Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2430 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2430/ ) HADOOP-8433 . Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2377 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2377/)
        HADOOP-8433. Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2377 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2377/ ) HADOOP-8433 . Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1075 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1075/)
        HADOOP-8433. Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1075 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1075/ ) HADOOP-8433 . Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1108 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1108/)
        HADOOP-8433. Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1108 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1108/ ) HADOOP-8433 . Don't set HADOOP_LOG_DIR in hadoop-env.sh. Contributed by Brahma Reddy Battula (Revision 1349641) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349641 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
        Arun C Murthy made changes -
        Affects Version/s 2.0.0-alpha [ 12320352 ]
        Affects Version/s 2.1.0-alpha [ 12321441 ]
        Fix Version/s 2.0.2-alpha [ 12322473 ]
        Fix Version/s 2.1.0-alpha [ 12321441 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1h 33m 1 Brahma Reddy Battula 25/May/12 04:55
        Patch Available Patch Available Resolved Resolved
        19d 1h 43m 1 Eli Collins 13/Jun/12 06:38
        Resolved Resolved Closed Closed
        120d 12h 6m 1 Arun C Murthy 11/Oct/12 18:45

          People

          • Assignee:
            Brahma Reddy Battula
            Reporter:
            Brahma Reddy Battula
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development