Hadoop Common
  1. Hadoop Common
  2. HADOOP-1795

Task.moveTaskOutputs is escaping special characters in output filenames

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.15.0
    • Component/s: None
    • Labels:
      None

      Description

      after a migration from 0.10.1 to 0.14.0, jobs can't generate output files with special characters in their name, just like '[' or ']' for example, because they are escaped during the Task.moveTaskOutputs process.

      For example, if you try to generate an output file named /foo/bar[0], it ends up being named /foo/bar%5B0%5B.

      The culprit is Task.getFinalPath(), when it does relativePath.toString(), where I think it should do relativePath.getPath().

      1. HADOOP-1795.patch
        6 kB
        Frédéric Bertin

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        18h 43m 1 Frédéric Bertin 29/Aug/07 14:09
        Patch Available Patch Available Resolved Resolved
        1d 8h 56m 1 Doug Cutting 30/Aug/07 23:05
        Resolved Resolved Closed Closed
        66d 20h 6m 1 Doug Cutting 05/Nov/07 18:12
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]
        Owen O'Malley made changes -
        Assignee Frédéric Bertin [ fred.bertin ]
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Dennis Kubes made changes -
        Comment [ This patch breaks the Injector job within Nutch.

        java.io.IOException: Target file:/c:/nutch/hadoop/mapred/temp/inject-temp-479521103/_reduce_xtsclf/part-00000 already exists
                at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:246)
                at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:125)
                at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:116)
                at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:180)
                at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:380)
                at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:452)
                at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:469)
                at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:426) ]
        Doug Cutting made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.15.0 [ 12312565 ]
        Resolution Fixed [ 1 ]
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, Frédéric!

        Show
        Doug Cutting added a comment - I just committed this. Thanks, Frédéric!
        Show
        Hadoop QA added a comment - +1 http://issues.apache.org/jira/secure/attachment/12364765/HADOOP-1795.patch applied and successfully tested against trunk revision r570881. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/646/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/646/console
        Frédéric Bertin made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Frédéric Bertin made changes -
        Field Original Value New Value
        Attachment HADOOP-1795.patch [ 12364765 ]
        Hide
        Frédéric Bertin added a comment -

        here it is (fix + test)

        Show
        Frédéric Bertin added a comment - here it is (fix + test)
        Hide
        Doug Cutting added a comment -

        It looks like this dates to HADOOP-1127, in 0.13.

        Can you supply a patch with a unit test? Thanks!

        Show
        Doug Cutting added a comment - It looks like this dates to HADOOP-1127 , in 0.13. Can you supply a patch with a unit test? Thanks!
        Frédéric Bertin created issue -

          People

          • Assignee:
            Frédéric Bertin
            Reporter:
            Frédéric Bertin
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development