Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14190

Copying folders containing = - characters between hdfs (using webhdfs) does not work in distcp

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.1
    • None
    • distcp
    • None

    Description

      Copying folders containing = - characters between hdfs (using webhdfs) does not work in distcp

      for example:

      src:hadoop2.7  target:hadoop3.1.1

      (1)

      hadoop distcp \
      -pugp \
      -i \
      webhdfs://1.1.1.1:50070/sudiyi_datawarehouse webhdfs://2.2.2.2:50070/sudiyi_datawarehouse

      ERROR tools.SimpleCopyListing: FileNotFoundException exception in listStatus: File /sudiyi_datawarehouse/st_device_standard_ds/date_time%3D2018-10-10 does not exist

       

      (2)

      hadoop distcp \
      -Dmapreduce.framework.name=yarn \
      -pugp \
      -i \
      webhdfs://1.1.1.1:50070/druid webhdfs://2.2.2.2:50070/druid

      Error: java.io.IOException: File copy failed: webhdfs://10.26.93.65:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800 --> webhdfs://10.27.234.198:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800

      at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:259)

      at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:217)

      at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48)

      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)

      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)

      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)

      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)

      at java.security.AccessController.doPrivileged(Native Method)

      at javax.security.auth.Subject.doAs(Subject.java:422)

      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)

      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

      Caused by: java.io.IOException: Couldn't run retriable-command: Copying webhdfs://10.26.93.65:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800 to webhdfs://10.27.234.198:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800

      at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)

      at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:256)

      ... 10 more

      Caused by: java.io.IOException: Failed to promote tmp-file:webhdfs://10.27.234.198:50070/druid/.distcp.tmp.attempt_1545990837043_0016_m_000015_2 to: webhdfs://10.27.234.198:50070/druid/indexing-logs/kill_task-myapp_V1-2018-04-26T16_20_55+0800

      at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:250)

      at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:140)

      at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)

      at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)

      Attachments

        Activity

          People

            aihuaxu Aihua Xu
            yinsong yinsong
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: