Hadoop Common
  1. Hadoop Common
  2. HADOOP-7438

Using the hadoop-deamon.sh script to start nodes leads to a depricated warning

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      hadoop-daemon.sh calls common/bin/hadoop for hdfs/bin/hdfs tasks and so common/bin/hadoop complains its deprecated for those uses.

      1. HADOOP-7438.2.patch
        1 kB
        Ravi Prakash
      2. HADOOP-7438.1.patch
        0.9 kB
        Ravi Prakash

        Activity

        Ravi Prakash created issue -
        Hide
        Ravi Prakash added a comment -

        Courtesy Arpit Gupta

        hadoop-daemon.sh start namenode
        is being issued to start a namenode.

        leads to the following warning

        DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.

        its also true when starting datanode. the warning is not there when using the same script to stop datanode

        Show
        Ravi Prakash added a comment - Courtesy Arpit Gupta hadoop-daemon.sh start namenode is being issued to start a namenode. leads to the following warning DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. its also true when starting datanode. the warning is not there when using the same script to stop datanode
        Hide
        Ravi Prakash added a comment -

        Simple enough patch. Not including any tests because I am not aware of any test framework.

        Show
        Ravi Prakash added a comment - Simple enough patch. Not including any tests because I am not aware of any test framework.
        Ravi Prakash made changes -
        Field Original Value New Value
        Attachment HADOOP-7438.1.patch [ 12484941 ]
        Hide
        Eric Yang added a comment -

        HADOOP_HDFS_HOME is for developer only to work on common/hdfs/mapreduce code in 3 separated source tree. HADOOP_HDFS_HOME should not be defined for production environment, hence reference to HADOOP_HDFS_HOME/bin/hdfs, should be HADOOP_PREFIX/bin/hdfs to be concise.

        Show
        Eric Yang added a comment - HADOOP_HDFS_HOME is for developer only to work on common/hdfs/mapreduce code in 3 separated source tree. HADOOP_HDFS_HOME should not be defined for production environment, hence reference to HADOOP_HDFS_HOME/bin/hdfs, should be HADOOP_PREFIX/bin/hdfs to be concise.
        Hide
        Ravi Prakash added a comment -

        Hi Eric,

        Thanks for your review. I'm afraid I am having some trouble starting hadoop from source. I have

        $ printenv | grep HADOOP
        HADOOP_HDFS_HOME=/home/raviprak/Code/hadoop/hadoop-common/hdfs
        HADOOP_COMMON_HOME=/home/raviprak/Code/hadoop/hadoop-common/common
        

        When I try to change the patch to have HADOOP_PREFIX instead of HADOOP_HDFS_HOME, this is what I get

        $ common/bin/hadoop-daemon.sh start namenode
        starting namenode, logging to /home/raviprak/Code/hadoop/hadoop-common/common/bin/../logs/hadoop-raviprak-namenode-localhost.out
        nice: /home/raviprak/Code/hadoop/hadoop-common/common/bin/../bin/hdfs: No such file or directory
        
        Show
        Ravi Prakash added a comment - Hi Eric, Thanks for your review. I'm afraid I am having some trouble starting hadoop from source. I have $ printenv | grep HADOOP HADOOP_HDFS_HOME=/home/raviprak/Code/hadoop/hadoop-common/hdfs HADOOP_COMMON_HOME=/home/raviprak/Code/hadoop/hadoop-common/common When I try to change the patch to have HADOOP_PREFIX instead of HADOOP_HDFS_HOME, this is what I get $ common/bin/hadoop-daemon.sh start namenode starting namenode, logging to /home/raviprak/Code/hadoop/hadoop-common/common/bin/../logs/hadoop-raviprak-namenode-localhost.out nice: /home/raviprak/Code/hadoop/hadoop-common/common/bin/../bin/hdfs: No such file or directory
        Hide
        Eric Yang added a comment -

        The recent changes is to ensure that hadoop-common, hadoop-hdfs, hadoop-mapreduce are installed to the same location. i.e. hadoop-[common|hdfs|mapreduce]-0.23.0/* move to /usr directory. This may not seem intuitive if user decided to keep hadoop bits in three separated directories. How about we added a IF condition?

        How about we check for existence of HADOOP_HDFS_HOME variable? if this is defined, use HADOOP_HDFS_HOME/bin/hdfs. If it does not exist, then fallback to HADOOP_PREFIX/bin/hdfs. This will ensure the script works for separated directories and as merged directory.

        Show
        Eric Yang added a comment - The recent changes is to ensure that hadoop-common, hadoop-hdfs, hadoop-mapreduce are installed to the same location. i.e. hadoop- [common|hdfs|mapreduce] -0.23.0/* move to /usr directory. This may not seem intuitive if user decided to keep hadoop bits in three separated directories. How about we added a IF condition? How about we check for existence of HADOOP_HDFS_HOME variable? if this is defined, use HADOOP_HDFS_HOME/bin/hdfs. If it does not exist, then fallback to HADOOP_PREFIX/bin/hdfs. This will ensure the script works for separated directories and as merged directory.
        Hide
        Ravi Prakash added a comment -

        Hi Eric,

        Thanks again for your review! I'm afraid I'm still trying to understand your suggestion. Lines 274-280 in hadoop-config.sh define HADOOP_HDFS_HOME relative to HADOOP_PREFIX when it is not defined

        # put hdfs in classpath if present
        if [ "$HADOOP_HDFS_HOME" = "" ]; then
          if [ -d "${HADOOP_PREFIX}/share/hadoop/hdfs" ]; then
            HADOOP_HDFS_HOME=$HADOOP_PREFIX/share/hadoop/hdfs
            #echo Found HDFS installed at $HADOOP_HDFS_HOME
          fi
        fi
        

        So HADOOP_HDFS_HOME would get defined when it is not present. Wouldn't it be better to define HADOOP_COMMON_HOME, HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME relative to HADOOP_PREFIX just once in hadoop-config, so that we don't have to use if-else blocks everywhere else?

        Show
        Ravi Prakash added a comment - Hi Eric, Thanks again for your review! I'm afraid I'm still trying to understand your suggestion. Lines 274-280 in hadoop-config.sh define HADOOP_HDFS_HOME relative to HADOOP_PREFIX when it is not defined # put hdfs in classpath if present if [ "$HADOOP_HDFS_HOME" = "" ]; then if [ -d "${HADOOP_PREFIX}/share/hadoop/hdfs" ]; then HADOOP_HDFS_HOME=$HADOOP_PREFIX/share/hadoop/hdfs #echo Found HDFS installed at $HADOOP_HDFS_HOME fi fi So HADOOP_HDFS_HOME would get defined when it is not present. Wouldn't it be better to define HADOOP_COMMON_HOME, HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME relative to HADOOP_PREFIX just once in hadoop-config, so that we don't have to use if-else blocks everywhere else?
        Hide
        Eric Yang added a comment -

        The intent is to minimize the number of environment variable that we use for production environment.

        Regards,
        Eric

        Show
        Eric Yang added a comment - The intent is to minimize the number of environment variable that we use for production environment. Regards, Eric
        Hide
        Ravi Prakash added a comment -

        Hi Eric,

        Thanks for your explanation. I've incorporated your comment and updated the patch. Could you please review and commit it?

        Thanks
        Ravi

        Show
        Ravi Prakash added a comment - Hi Eric, Thanks for your explanation. I've incorporated your comment and updated the patch. Could you please review and commit it? Thanks Ravi
        Ravi Prakash made changes -
        Attachment HADOOP-7438.2.patch [ 12485755 ]
        Ravi Prakash made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Eric Yang added a comment -

        +1 looks good. Thanks Ravi.

        Show
        Eric Yang added a comment - +1 looks good. Thanks Ravi.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12485755/HADOOP-7438.2.patch
        against trunk revision 1147971.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/744//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/744//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/744//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485755/HADOOP-7438.2.patch against trunk revision 1147971. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/744//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/744//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/744//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Ravi, is this for 0.22 only?

        Show
        Suresh Srinivas added a comment - Ravi, is this for 0.22 only?
        Hide
        Ravi Prakash added a comment -

        Hi Suresh,

        Yes! Its tagged 0.22.100.

        Thanks
        Ravi.

        Show
        Ravi Prakash added a comment - Hi Suresh, Yes! Its tagged 0.22.100. Thanks Ravi.
        Hide
        Ravi Prakash added a comment -

        Hi Suresh, To clarify, this needs to go into trunk. Also adding fix version to 0.23

        Show
        Ravi Prakash added a comment - Hi Suresh, To clarify, this needs to go into trunk. Also adding fix version to 0.23
        Ravi Prakash made changes -
        Fix Version/s 0.23.0 [ 12315569 ]
        Hide
        Suresh Srinivas added a comment -

        0.22.100 is a Yahoo label. The correct fix version is 0.23. +1 for the patch.

        Show
        Suresh Srinivas added a comment - 0.22.100 is a Yahoo label. The correct fix version is 0.23. +1 for the patch.
        Hide
        Suresh Srinivas added a comment -

        I committed the patch. Thank you Ravi.

        Show
        Suresh Srinivas added a comment - I committed the patch. Thank you Ravi.
        Suresh Srinivas made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Ravi Prakash added a comment -

        Thanks tons Suresh!

        Show
        Ravi Prakash added a comment - Thanks tons Suresh!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #699 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/699/)
        HADOOP-7438. Fix deprecated warnings from hadoop-daemon.sh script. Contributed by Ravi Prakash.

        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150914
        Files :

        • /hadoop/common/trunk/common/CHANGES.txt
        • /hadoop/common/trunk/common/bin/hadoop-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #699 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/699/ ) HADOOP-7438 . Fix deprecated warnings from hadoop-daemon.sh script. Contributed by Ravi Prakash. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150914 Files : /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/bin/hadoop-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #758 (See https://builds.apache.org/job/Hadoop-Common-trunk/758/)
        HADOOP-7438. Fix deprecated warnings from hadoop-daemon.sh script. Contributed by Ravi Prakash.

        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150914
        Files :

        • /hadoop/common/trunk/common/CHANGES.txt
        • /hadoop/common/trunk/common/bin/hadoop-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #758 (See https://builds.apache.org/job/Hadoop-Common-trunk/758/ ) HADOOP-7438 . Fix deprecated warnings from hadoop-daemon.sh script. Contributed by Ravi Prakash. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150914 Files : /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/bin/hadoop-daemon.sh
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Ravi Prakash
            Reporter:
            Ravi Prakash
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development