[HDFS-9670] DistCp throws NPE when source is root

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: distcp
• Labels: supportability
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: John Zhuge
• Reporter: Yongjun Zhang
• Created: 20/Jan/16 20:26
• Updated: 30/Aug/16 01:20
• Resolved: 21/Apr/16 19:36

Description

Symptom:

[root@vb0724 ~]# hadoop distcp hdfs://X:8020/ hdfs://Y:8020/
16/01/20 11:33:33 INFO tools.DistCp: Input Options: DistCpOptions

Unknown macro: {atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs}

16/01/20 11:33:33 INFO client.RMProxy: Connecting to ResourceManager at Z:8032
16/01/20 11:33:33 ERROR tools.DistCp: Exception encountered
java.lang.NullPointerException
at org.apache.hadoop.tools.util.DistCpUtils.getRelativePath(DistCpUtils.java:144)
at org.apache.hadoop.tools.SimpleCopyListing.writeToFileListing(SimpleCopyListing.java:598)
at org.apache.hadoop.tools.SimpleCopyListing.writeToFileListingRoot(SimpleCopyListing.java:583)
at org.apache.hadoop.tools.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:313)
at org.apache.hadoop.tools.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:174)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:365)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:171)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)

Relevant code:

  private Path computeSourceRootPath(FileStatus sourceStatus,
                                     DistCpOptions options) throws IOException {

    Path target = options.getTargetPath();
    FileSystem targetFS = target.getFileSystem(getConf());
    final boolean targetPathExists = options.getTargetPathExists();

    boolean solitaryFile = options.getSourcePaths().size() == 1
                                                && !sourceStatus.isDirectory();

    if (solitaryFile) {
      if (targetFS.isFile(target) || !targetPathExists) {
        return sourceStatus.getPath();
      } else {
        return sourceStatus.getPath().getParent();
      }
    } else {
      boolean specialHandling = (options.getSourcePaths().size() == 1 && !targetPathExists) ||
          options.shouldSyncFolder() || options.shouldOverwrite();

      return specialHandling && sourceStatus.isDirectory() ? sourceStatus.getPath() :
          sourceStatus.getPath().getParent();
    }
  }

We can see that it could return NULL at the end when doing sourceStatus.getPath().getParent()

Issue Links

• is depended upon by: HDFS-10300 TestDistCpSystem should share MiniDFSCluster