Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9670

DistCp throws NPE when source is root

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Symptom:

      [root@vb0724 ~]# hadoop distcp hdfs://X:8020/ hdfs://Y:8020/
      16/01/20 11:33:33 INFO tools.DistCp: Input Options: DistCpOptions

      Unknown macro: {atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs}

      16/01/20 11:33:33 INFO client.RMProxy: Connecting to ResourceManager at Z:8032
      16/01/20 11:33:33 ERROR tools.DistCp: Exception encountered
      java.lang.NullPointerException
      at org.apache.hadoop.tools.util.DistCpUtils.getRelativePath(DistCpUtils.java:144)
      at org.apache.hadoop.tools.SimpleCopyListing.writeToFileListing(SimpleCopyListing.java:598)
      at org.apache.hadoop.tools.SimpleCopyListing.writeToFileListingRoot(SimpleCopyListing.java:583)
      at org.apache.hadoop.tools.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:313)
      at org.apache.hadoop.tools.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:174)
      at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
      at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
      at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
      at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:365)
      at org.apache.hadoop.tools.DistCp.execute(DistCp.java:171)
      at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)

      Relevant code:

        private Path computeSourceRootPath(FileStatus sourceStatus,
                                           DistCpOptions options) throws IOException {
      
          Path target = options.getTargetPath();
          FileSystem targetFS = target.getFileSystem(getConf());
          final boolean targetPathExists = options.getTargetPathExists();
      
          boolean solitaryFile = options.getSourcePaths().size() == 1
                                                      && !sourceStatus.isDirectory();
      
          if (solitaryFile) {
            if (targetFS.isFile(target) || !targetPathExists) {
              return sourceStatus.getPath();
            } else {
              return sourceStatus.getPath().getParent();
            }
          } else {
            boolean specialHandling = (options.getSourcePaths().size() == 1 && !targetPathExists) ||
                options.shouldSyncFolder() || options.shouldOverwrite();
      
            return specialHandling && sourceStatus.isDirectory() ? sourceStatus.getPath() :
                sourceStatus.getPath().getParent();
          }
        }
      

      We can see that it could return NULL at the end when doing sourceStatus.getPath().getParent()

      Attachments

        1. HDFS-9670.002.patch
          4 kB
          John Zhuge
        2. HDFS-9670.001.patch
          4 kB
          John Zhuge

        Issue Links

          Activity

            People

              jzhuge John Zhuge
              yzhangal Yongjun Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: