Hadoop Common
  1. Hadoop Common
  2. HADOOP-4717

Removal of default port# in NameNode.getUri() cause a map/reduce job failed to prompt temporay output

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Problem reported here is that when the default port number (8020) is specified in the output, job succeeds but no output is created. The cause of the problem is that "listStatus" call drops the port number because NameNode.getUri removes the default port#.

      Assuming that a map/reduce output directory is set to be "hdfs://localhost:8020/out", A call "listStatus" on any of its sub directory, for example, "hdfs://localhost:8020/out/tempXX", returns results like below:

      hdfs://localhost/out/tempXX/part-00005

      Because of this, Task.java
      574 private Path getFinalPath(Path jobOutputDir, Path taskOutput) {
      575 URI relativePath = taskOutputPath.toUri().relativize(taskOutput.toUri());

      does not get the correct relativePath because TaskOutputPath contain ports, but taskOutput doesn't.

      It seems to me that the problem could be fixed if we make Path.makeQualified() to return the same path not matter the input path contains the default port or not.

      1. HADOOP-4717.patch
        1 kB
        Doug Cutting
      2. relativePath.patch
        4 kB
        Hairong Kuang
      3. relativePath1.patch
        4 kB
        Hairong Kuang

        Issue Links

          Activity

          Koji Noguchi made changes -
          Link This issue relates to MAPREDUCE-837 [ MAPREDUCE-837 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hairong Kuang made changes -
          Resolution Fixed [ 1 ]
          Hadoop Flags [Reviewed]
          Assignee Doug Cutting [ cutting ] Hairong Kuang [ hairong ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hairong Kuang made changes -
          Attachment relativePath1.patch [ 12395320 ]
          Hairong Kuang made changes -
          Attachment relativePath.patch [ 12395242 ]
          Hairong Kuang made changes -
          Assignee Hairong Kuang [ hairong ] Doug Cutting [ cutting ]
          Hairong Kuang made changes -
          Link This issue is blocked by HADOOP-4746 [ HADOOP-4746 ]
          Nigel Daley made changes -
          Assignee Hairong Kuang [ hairong ]
          Nigel Daley made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Doug Cutting made changes -
          Field Original Value New Value
          Attachment HADOOP-4717.patch [ 12394705 ]
          Hairong Kuang created issue -

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development