Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3100

distcp with an hftp destination url fails if the destination directory does not exist

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: None
    • Component/s: distcp
    • Labels:
      None

      Description

      bash-3.2$ bin/hadoop --config HADOOP_CONF_DIR distcp -i hftp://NN_HOST:50070/input_path hftp://NN_HOST:50070/output_path
      11/09/27 04:36:01 INFO tools.DistCp: srcPaths=[hftp://hftp://NN_HOST:50070/input_path]
      11/09/27 04:36:01 INFO tools.DistCp: destPath=hftp://hftp://NN_HOST:50070/output_path
      11/09/27 04:36:02 INFO tools.DistCp: hftp://hftp://NN_HOST:50070/output_path does not exist.
      With failures, global counters are inaccurate; consider running with -i
      Copy failed: java.io.IOException: Not supported
      at org.apache.hadoop.hdfs.HftpFileSystem.mkdirs(HftpFileSystem.java:558)
      at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1066)
      at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1170)
      at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
      at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)

      The same command works if instead of hftp default filesystem (hdfs) is used. It creates the dir if it does not exist. We should do the same for hftp. I also suspect that we have this issue with webhdfs.

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Arpit Gupta
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development