Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Symptom:

      [root@vb0724 ~]# hadoop distcp hdfs://X:8020/ hdfs://Y:8020/
      16/01/20 11:33:33 INFO tools.DistCp: Input Options: DistCpOptions

      Unknown macro: {atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs}

      16/01/20 11:33:33 INFO client.RMProxy: Connecting to ResourceManager at Z:8032
      16/01/20 11:33:33 ERROR tools.DistCp: Exception encountered
      java.lang.NullPointerException
      at org.apache.hadoop.tools.util.DistCpUtils.getRelativePath(DistCpUtils.java:144)
      at org.apache.hadoop.tools.SimpleCopyListing.writeToFileListing(SimpleCopyListing.java:598)
      at org.apache.hadoop.tools.SimpleCopyListing.writeToFileListingRoot(SimpleCopyListing.java:583)
      at org.apache.hadoop.tools.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:313)
      at org.apache.hadoop.tools.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:174)
      at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
      at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
      at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
      at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:365)
      at org.apache.hadoop.tools.DistCp.execute(DistCp.java:171)
      at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)

      Relevant code:

        private Path computeSourceRootPath(FileStatus sourceStatus,
                                           DistCpOptions options) throws IOException {
      
          Path target = options.getTargetPath();
          FileSystem targetFS = target.getFileSystem(getConf());
          final boolean targetPathExists = options.getTargetPathExists();
      
          boolean solitaryFile = options.getSourcePaths().size() == 1
                                                      && !sourceStatus.isDirectory();
      
          if (solitaryFile) {
            if (targetFS.isFile(target) || !targetPathExists) {
              return sourceStatus.getPath();
            } else {
              return sourceStatus.getPath().getParent();
            }
          } else {
            boolean specialHandling = (options.getSourcePaths().size() == 1 && !targetPathExists) ||
                options.shouldSyncFolder() || options.shouldOverwrite();
      
            return specialHandling && sourceStatus.isDirectory() ? sourceStatus.getPath() :
                sourceStatus.getPath().getParent();
          }
        }
      

      We can see that it could return NULL at the end when doing sourceStatus.getPath().getParent()

      1. HDFS-9670.001.patch
        4 kB
        John Zhuge
      2. HDFS-9670.002.patch
        4 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9647 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9647/)
          HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via (yzhang: rev a749ba0ceaa843aa83146b6bea19e031c8dc3296)

          • hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
          • hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9647 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9647/ ) HDFS-9670 . DistCp throws NPE when source is root. (John Zhuge via (yzhang: rev a749ba0ceaa843aa83146b6bea19e031c8dc3296) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSystem.java
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Yongjun Zhang for reporting, reviewing, and committing the jira.

          Show
          jzhuge John Zhuge added a comment - Thanks Yongjun Zhang for reporting, reviewing, and committing the jira.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Committed to trunk, branch-2, branch-2.8.

          Thanks John Zhuge much for the contribution!

          Show
          yzhangal Yongjun Zhang added a comment - Committed to trunk, branch-2, branch-2.8. Thanks John Zhuge much for the contribution!
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks for the new rev John Zhuge,

          LGTM, +1 and I will commit soon. Sorry for the delayed review.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks for the new rev John Zhuge , LGTM, +1 and I will commit soon. Sorry for the delayed review.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 8s trunk passed
          +1 compile 0m 19s trunk passed with JDK v1.8.0_77
          +1 compile 0m 16s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 0m 14s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.8.0_77
          +1 javac 0m 16s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.7.0_95
          +1 javac 0m 16s the patch passed
          +1 checkstyle 0m 10s hadoop-tools/hadoop-distcp: patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19)
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 41s the patch passed
          +1 javadoc 0m 14s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          +1 unit 8m 32s hadoop-distcp in the patch passed with JDK v1.8.0_77.
          +1 unit 7m 25s hadoop-distcp in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          29m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799010/HDFS-9670.002.patch
          JIRA Issue HDFS-9670
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux db40902e1ce5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 69f3d42
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15173/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15173/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 8s trunk passed +1 compile 0m 19s trunk passed with JDK v1.8.0_77 +1 compile 0m 16s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 18s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_77 +1 javac 0m 16s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_95 +1 javac 0m 16s the patch passed +1 checkstyle 0m 10s hadoop-tools/hadoop-distcp: patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 8m 32s hadoop-distcp in the patch passed with JDK v1.8.0_77. +1 unit 7m 25s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 29m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799010/HDFS-9670.002.patch JIRA Issue HDFS-9670 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux db40902e1ce5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 69f3d42 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15173/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15173/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Created HDFS-10300 "TestDistCpSystem should share MiniDFSCluster".

          Show
          jzhuge John Zhuge added a comment - Created HDFS-10300 "TestDistCpSystem should share MiniDFSCluster".
          Hide
          jzhuge John Zhuge added a comment -

          Patch 002:

          • Use 1 mini cluster in unit test because the issue is reproducible with both source and target on the same cluster.
          Show
          jzhuge John Zhuge added a comment - Patch 002: Use 1 mini cluster in unit test because the issue is reproducible with both source and target on the same cluster.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks, will fix it.

          Show
          jzhuge John Zhuge added a comment - Thanks, will fix it.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi John Zhuge,

          For the new test you added, it seems creating one cluster would be sufficient. Would you please look into?

          Then we can consider a future jira for consolidating the set of tests.

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi John Zhuge , For the new test you added, it seems creating one cluster would be sufficient. Would you please look into? Then we can consider a future jira for consolidating the set of tests. Thanks.
          Hide
          jzhuge John Zhuge added a comment -

          Very good point!

          • File a separate jira to consolidate mini cluster in this test class?
          • Or bundle the change in this patch?
          Show
          jzhuge John Zhuge added a comment - Very good point! File a separate jira to consolidate mini cluster in this test class? Or bundle the change in this patch?
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi John Zhuge,

          Thanks for working on this issue. The solution looks good to me. One comment about the test code here. The cost of starting Mini cluster is expensive, ideally we could try to think about using the same cluster for the set of tests. In this case, can we at least try to create a single cluster and do distcp within the same cluster?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi John Zhuge , Thanks for working on this issue. The solution looks good to me. One comment about the test code here. The cost of starting Mini cluster is expensive, ideally we could try to think about using the same cluster for the set of tests. In this case, can we at least try to create a single cluster and do distcp within the same cluster? Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 22s trunk passed
          +1 compile 0m 17s trunk passed with JDK v1.8.0_77
          +1 compile 0m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 31s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_77
          +1 javac 0m 12s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_95
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 10s hadoop-tools/hadoop-distcp: patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19)
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 34s the patch passed
          +1 javadoc 0m 9s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          +1 unit 8m 21s hadoop-distcp in the patch passed with JDK v1.8.0_77.
          +1 unit 7m 20s hadoop-distcp in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          29m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798664/HDFS-9670.001.patch
          JIRA Issue HDFS-9670
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 282fa29c44ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27b131e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15161/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15161/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 22s trunk passed +1 compile 0m 17s trunk passed with JDK v1.8.0_77 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 31s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 17s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_77 +1 javac 0m 12s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_95 +1 javac 0m 14s the patch passed +1 checkstyle 0m 10s hadoop-tools/hadoop-distcp: patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19) +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 9s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 8m 21s hadoop-distcp in the patch passed with JDK v1.8.0_77. +1 unit 7m 20s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 29m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798664/HDFS-9670.001.patch JIRA Issue HDFS-9670 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 282fa29c44ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27b131e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15161/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15161/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 001:

          • computeSourceRootPath returns itself instead of its parent when source is root.
          • Add unit test TestDistCpSystem#testSourceRoot
          Show
          jzhuge John Zhuge added a comment - Patch 001: computeSourceRootPath returns itself instead of its parent when source is root. Add unit test TestDistCpSystem#testSourceRoot
          Hide
          jzhuge John Zhuge added a comment -

          Source file system can also be different, e.g., s3a.

          Show
          jzhuge John Zhuge added a comment - Source file system can also be different, e.g., s3a .
          Hide
          jzhuge John Zhuge added a comment -

          Reproduced test in different ways:

          • The target dir must exist.
          • The target can be:
            • on a different cluster (the jira test case),
            • or on the same cluster,
            • or on a different file system, e.g., s3a
          Show
          jzhuge John Zhuge added a comment - Reproduced test in different ways: The target dir must exist. The target can be: on a different cluster (the jira test case), or on the same cluster, or on a different file system, e.g., s3a
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks John for looking into.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks John for looking into.

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              yzhangal Yongjun Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development