Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4717

Removal of default port# in NameNode.getUri() cause a map/reduce job failed to prompt temporay output

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Problem reported here is that when the default port number (8020) is specified in the output, job succeeds but no output is created. The cause of the problem is that "listStatus" call drops the port number because NameNode.getUri removes the default port#.

      Assuming that a map/reduce output directory is set to be "hdfs://localhost:8020/out", A call "listStatus" on any of its sub directory, for example, "hdfs://localhost:8020/out/tempXX", returns results like below:

      hdfs://localhost/out/tempXX/part-00005

      Because of this, Task.java
      574 private Path getFinalPath(Path jobOutputDir, Path taskOutput) {
      575 URI relativePath = taskOutputPath.toUri().relativize(taskOutput.toUri());

      does not get the correct relativePath because TaskOutputPath contain ports, but taskOutput doesn't.

      It seems to me that the problem could be fixed if we make Path.makeQualified() to return the same path not matter the input path contains the default port or not.

        Attachments

        1. relativePath1.patch
          4 kB
          Hairong Kuang
        2. relativePath.patch
          4 kB
          Hairong Kuang
        3. HADOOP-4717.patch
          1 kB
          Doug Cutting

          Issue Links

            Activity

              People

              • Assignee:
                hairong Hairong Kuang
                Reporter:
                hairong Hairong Kuang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: