Hadoop Common
  1. Hadoop Common
  2. HADOOP-6630

hadoop-config.sh fails to get executed if hadoop wrapper scripts are in path

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If the hadoop/bin commands are in the path, hadoop-config.sh doesn't executed until much later in the stack because $0 lacks a path.

      1. HADOOP-6630.txt
        4 kB
        Allen Wittenauer

        Activity

        Hide
        Allen Wittenauer added a comment -

        Since we require bash, we can cheat a bit. BASH_SOURCE should contain the full path to the script that we are executing. This means we can piggyback off of that variable to figure out where our real landing path is at.

        Show
        Allen Wittenauer added a comment - Since we require bash, we can cheat a bit. BASH_SOURCE should contain the full path to the script that we are executing. This means we can piggyback off of that variable to figure out where our real landing path is at.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12438560/HADOOP-6630.txt
        against trunk revision 921980.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/35/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/35/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/35/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/35/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12438560/HADOOP-6630.txt against trunk revision 921980. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/35/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/35/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/35/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/35/console This message is automatically generated.
        Hide
        Allen Wittenauer added a comment -

        For such a simple change, I doubt tests are required.

        Show
        Allen Wittenauer added a comment - For such a simple change, I doubt tests are required.
        Hide
        Doug Cutting added a comment -

        Allen, if BASH_SOURCE contains the full path, then can we get rid of the 'bin=`cd "$bin"; pwd`' that follows each of these, since the point of that is to get the full path of the bin directory?

        Show
        Doug Cutting added a comment - Allen, if BASH_SOURCE contains the full path, then can we get rid of the 'bin=`cd "$bin"; pwd`' that follows each of these, since the point of that is to get the full path of the bin directory?
        Hide
        Allen Wittenauer added a comment -

        Actually, I misspoke a bit. It will be full path if it comes from $PATH. If you call with a relative path, it will contain the relative path.

        So to be more explicit:

        if I put this code:

        #!/bin/bash
        echo $BASH_SOURCE
        

        in $HOME/bin and make sure $HOME/bin is in my path, you'll see the following:

        awittena-mn:bin awittena$ testexec
        /Users/awittena/bin/testexec

        awittena-mn:~ awittena$ bin/testexec
        bin/testexec

        awittena-mn:~ awittena$ cd bin && testexec
        /Users/awittena/bin/testexec

        awittena-mn:~ awittena$ cd bin && ./testexec
        ./testexec

        So the cd+pwd combo is still needed, but this should fix the "in the $PATH" issue. It is a shame that bash hid the "can't find hadoop-config.sh" error for the past few years. If you run it with -x set, you'll see it tho.

        Show
        Allen Wittenauer added a comment - Actually, I misspoke a bit. It will be full path if it comes from $PATH. If you call with a relative path, it will contain the relative path. So to be more explicit: if I put this code: #!/bin/bash echo $BASH_SOURCE in $HOME/bin and make sure $HOME/bin is in my path, you'll see the following: awittena-mn:bin awittena$ testexec /Users/awittena/bin/testexec awittena-mn:~ awittena$ bin/testexec bin/testexec awittena-mn:~ awittena$ cd bin && testexec /Users/awittena/bin/testexec awittena-mn:~ awittena$ cd bin && ./testexec ./testexec So the cd+pwd combo is still needed, but this should fix the "in the $PATH" issue. It is a shame that bash hid the "can't find hadoop-config.sh" error for the past few years. If you run it with -x set, you'll see it tho.
        Hide
        Doug Cutting added a comment -

        Sounds reasonable. Thanks for the explanation. +1 from me.

        Show
        Doug Cutting added a comment - Sounds reasonable. Thanks for the explanation. +1 from me.
        Hide
        Allen Wittenauer added a comment -

        Can we get this committed prior to the 0.21 freeze please?

        Thank you.

        Show
        Allen Wittenauer added a comment - Can we get this committed prior to the 0.21 freeze please? Thank you.
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Allen!

        Show
        Tom White added a comment - I've just committed this. Thanks Allen!

          People

          • Assignee:
            Allen Wittenauer
            Reporter:
            Allen Wittenauer
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development