Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1285

DistCp cannot handle -delete if destination is local filesystem

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.21.0
    • Component/s: distcp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following exception is thrown:

      Copy failed: java.io.IOException: wrong value class: org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus is not class org.apache.hadoop.fs.FileStatus
      	at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:988)
      	at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:977)
      	at org.apache.hadoop.tools.DistCp.deleteNonexisting(DistCp.java:1226)
      	at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1134)
      	at org.apache.hadoop.tools.DistCp.copy(DistCp.java:650)
      	at org.apache.hadoop.tools.DistCp.run(DistCp.java:857)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      
      1. MAPREDUCE-1285.patch
        3 kB
        Peter Romianowski
      2. MAPREDUCE-1285-trunk.patch
        3 kB
        Peter Romianowski

        Activity

        Hide
        Tsz Wo Nicholas Sze added a comment -

        Interestingly, there is a comment saying that RawLocalFileStatus would break DistCp.FilePair. (Note that CopyFiles is a former name of DistCp.)

          static class RawLocalFileStatus extends FileStatus {
            /* We can add extra fields here. It breaks at least CopyFiles.FilePair().
             * We recognize if the information is already loaded by check if
             * onwer.equals("").
             */
        
        Show
        Tsz Wo Nicholas Sze added a comment - Interestingly, there is a comment saying that RawLocalFileStatus would break DistCp.FilePair. (Note that CopyFiles is a former name of DistCp.) static class RawLocalFileStatus extends FileStatus { /* We can add extra fields here. It breaks at least CopyFiles.FilePair(). * We recognize if the information is already loaded by check if * onwer.equals(""). */
        Hide
        Peter Romianowski added a comment -

        I don't think that the patch breaks anything at this place. I only changed a single line in deleteNonexisting() where FilePair is not even used. The problem has been that the SequenceFile.Writer was tied to FileStatus.class but it didn't have to since we know the concrete implementation.

        So the patch should be save even if someone introduces new fields in RawLocalFileStatus.

        Did I miss something?

        Show
        Peter Romianowski added a comment - I don't think that the patch breaks anything at this place. I only changed a single line in deleteNonexisting() where FilePair is not even used. The problem has been that the SequenceFile.Writer was tied to FileStatus.class but it didn't have to since we know the concrete implementation. So the patch should be save even if someone introduces new fields in RawLocalFileStatus. Did I miss something?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi Peter, I am not saying that your patch is breaking anything. I mean that when RawLocalFileStatus is created, it is known that RawLocalFileStatus would break DistCp.FilePair.

        It seems that your patch is not generated from trunk since the line numbers do not match. Could you generate a new patch from trunk?

        Show
        Tsz Wo Nicholas Sze added a comment - Hi Peter, I am not saying that your patch is breaking anything. I mean that when RawLocalFileStatus is created, it is known that RawLocalFileStatus would break DistCp.FilePair. It seems that your patch is not generated from trunk since the line numbers do not match. Could you generate a new patch from trunk?
        Hide
        Peter Romianowski added a comment -

        Patch is against 0.20.1 (git://github.com/optivo/hadoop-0.20.1.git). I'll add one for trunk.

        Show
        Peter Romianowski added a comment - Patch is against 0.20.1 (git://github.com/optivo/hadoop-0.20.1.git). I'll add one for trunk.
        Hide
        Peter Romianowski added a comment -

        Added patch against trunk.

        Show
        Peter Romianowski added a comment - Added patch against trunk.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427517/MAPREDUCE-1285-trunk.patch
        against trunk revision 888761.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/178/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/178/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/178/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/178/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427517/MAPREDUCE-1285-trunk.patch against trunk revision 888761. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/178/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/178/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/178/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/178/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427517/MAPREDUCE-1285-trunk.patch
        against trunk revision 888761.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/179/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/179/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/179/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/179/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427517/MAPREDUCE-1285-trunk.patch against trunk revision 888761. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/179/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/179/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/179/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/179/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Peter!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Peter!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #171 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/171/)
        . In DistCp.deleteNonexisting(..), get class from the parameter instead of using FileStatus.class. Contributed by Peter Romianowski

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #171 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/171/ ) . In DistCp.deleteNonexisting(..), get class from the parameter instead of using FileStatus.class. Contributed by Peter Romianowski

          People

          • Assignee:
            Peter Romianowski
            Reporter:
            Peter Romianowski
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development