Hadoop Common
  1. Hadoop Common
  2. HADOOP-2855

[HOD] HOD fails to allocate a cluster if the tarball specified is a relative path

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changes were made to handle relative paths correctly for important HOD options such as the cluster directory, tarball option, and script file.

      Description

      Run hod -t my-tar.tar.gz -o "allocate hod-cluster 3". Ringmaster fails to come up. The log shows the exception as an invalid URL for the tarball file. Basically HOD should translate the relative path to an absolute path and send it to the ringmaster.

      1. HADOOP-2855
        7 kB
        Vinod Kumar Vavilapalli
      2. HADOOP-2855.1
        15 kB
        Vinod Kumar Vavilapalli
      3. HADOOP-2855.2
        15 kB
        Vinod Kumar Vavilapalli

        Activity

        Hide
        Vinod Kumar Vavilapalli added a comment -

        Earlier behavior was not to normalize any paths(files or directories), so relative paths weren't getting expanded. Changed that behavior now. Tested with relative paths for ringmaster.hadoop-tar-ball, hod.script and hod.clusterdir.

        Also added a new data type uri. Ringmaster.hadoop-tar-ball is changed to be of this data type. So, a URI of the format file:// can be used to denote a path on the local file system. hodring.log-destination-uri also needs to changed to use this data type. Will file a jira for that.

        Attaching patch.

        Show
        Vinod Kumar Vavilapalli added a comment - Earlier behavior was not to normalize any paths(files or directories), so relative paths weren't getting expanded. Changed that behavior now. Tested with relative paths for ringmaster.hadoop-tar-ball, hod.script and hod.clusterdir. Also added a new data type uri. Ringmaster.hadoop-tar-ball is changed to be of this data type. So, a URI of the format file:// can be used to denote a path on the local file system. hodring.log-destination-uri also needs to changed to use this data type. Will file a jira for that. Attaching patch.
        Hide
        Hemanth Yamijala added a comment -

        Blocker for 0.17. Users are facing this.

        Show
        Hemanth Yamijala added a comment - Blocker for 0.17. Users are facing this.
        Hide
        Hemanth Yamijala added a comment -

        Couple of comments:

        • Need unit test cases for validation and normalization of uri and directories.
        • We are validating the hadoop tar ball URI on the client. Since the current expectation is that the tarball exists only on the compute nodes, this may result in some regression - even though it is an unlikely scenario. Even if we keep this, the error message on validation failure is a bit confusing. It says invalid uri, but as far as the user is concerned it just seems like a file path to him.
        • For every datatype in types.py, there are 4 functions. The display_uri function is one of the 4 that's missing. Is this required ?
        • Do we need to do expanduser in types.py: __normalizePath ?
        • The code for verify_uri and norm_uri is trying to normalize a path even if the uri begins with file://. If we specify the URI in this format, I don't think the path can be relative, right ? So, I think we should only be checking for empty string, and not 'file' for the value of component[0].
        Show
        Hemanth Yamijala added a comment - Couple of comments: Need unit test cases for validation and normalization of uri and directories. We are validating the hadoop tar ball URI on the client. Since the current expectation is that the tarball exists only on the compute nodes, this may result in some regression - even though it is an unlikely scenario. Even if we keep this, the error message on validation failure is a bit confusing. It says invalid uri, but as far as the user is concerned it just seems like a file path to him. For every datatype in types.py, there are 4 functions. The display_uri function is one of the 4 that's missing. Is this required ? Do we need to do expanduser in types.py: __normalizePath ? The code for verify_uri and norm_uri is trying to normalize a path even if the uri begins with file:// . If we specify the URI in this format, I don't think the path can be relative, right ? So, I think we should only be checking for empty string, and not 'file' for the value of component [0] .
        Hide
        Vinod Kumar Vavilapalli added a comment - - edited
        • Added test cases.
        • Removed validating the hadoop tar ball URI on the client. That wasn't intended even eariler, it just crept in while testing validation of uri data type.
        • Not adding the display_uri method. The class display itself is dead code.
        • Not expanding user (e.g ~ or ~user)in types.py. This is a shell feature, leaving it for the shell to expand.
        • We are normalizing paths given in file:// scheme also, to deal with relative paths like file://localhost/tmp/../tempDir etc.
        • Fixed a couple of bugs that test cases exposed.
          Eariler normalizing file:// type uri also used to return a normal system path. Changed that behaviour to return a normalize file:// schemed uri.
          Code to verify paths, other that system paths and file://, like those of scheme http:// for example, was missing earlier. Added that now.

        Attaching the patch.

        Show
        Vinod Kumar Vavilapalli added a comment - - edited Added test cases. Removed validating the hadoop tar ball URI on the client. That wasn't intended even eariler, it just crept in while testing validation of uri data type. Not adding the display_uri method. The class display itself is dead code. Not expanding user (e.g ~ or ~user)in types.py. This is a shell feature, leaving it for the shell to expand. We are normalizing paths given in file:// scheme also, to deal with relative paths like file://localhost/tmp/../tempDir etc. Fixed a couple of bugs that test cases exposed. Eariler normalizing file:// type uri also used to return a normal system path. Changed that behaviour to return a normalize file:// schemed uri. Code to verify paths, other that system paths and file:// , like those of scheme http:// for example, was missing earlier. Added that now. Attaching the patch.
        Hide
        Hemanth Yamijala added a comment -

        Code and tests looks good. Works with relative paths correctly.

        Could we have added a test for checking relative path expansion, that normalization actually expands the relative path ? Maybe create a tmp file in the current directory (which is the original directory), chdir to /tmp, normalize, verify that the tmp file is normalized to the full path in the original directory, then chdir back to the original directory. Would that work ?

        Apart from this, +1.

        Show
        Hemanth Yamijala added a comment - Code and tests looks good. Works with relative paths correctly. Could we have added a test for checking relative path expansion, that normalization actually expands the relative path ? Maybe create a tmp file in the current directory (which is the original directory), chdir to /tmp, normalize, verify that the tmp file is normalized to the full path in the original directory, then chdir back to the original directory. Would that work ? Apart from this, +1.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Incorporating Hemanth's suggestion. Rettaching the patch.

        Show
        Vinod Kumar Vavilapalli added a comment - Incorporating Hemanth's suggestion. Rettaching the patch.
        Hide
        Hemanth Yamijala added a comment -

        +1 for new changes.

        Show
        Hemanth Yamijala added a comment - +1 for new changes.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12378629/HADOOP-2855.2
        against trunk revision 619744.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 4 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2062/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2062/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2062/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2062/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378629/HADOOP-2855.2 against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 4 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2062/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2062/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2062/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2062/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12378629/HADOOP-2855.2
        against trunk revision 619744.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 4 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2065/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2065/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2065/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2065/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378629/HADOOP-2855.2 against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 4 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2065/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2065/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2065/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2065/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this. Thanks, Vinod !

        Show
        Hemanth Yamijala added a comment - I just committed this. Thanks, Vinod !

          People

          • Assignee:
            Vinod Kumar Vavilapalli
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development