[HDFS-16363] An exception occurs in the distcp task of a large number of files, when yarn.app.mapreduce.am.staging-dir is set to the hdfs path.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.2
• Component/s: distcp
• Assignee: Unassigned
• Reporter: Zhen Wang
• Created: 01/Dec/21 06:53
• Updated: 01/Dec/21 13:09

Description

An exception occurs in the distcp task of a large number of files, when yarn.app.mapreduce.am.staging-dir is set to the hdfs path.

 

task log:

21/12/01 13:56:04 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 24631997; dirCnt = 1750444
21/12/01 13:56:04 INFO tools.SimpleCopyListing: Build file listing completed.
21/12/01 13:56:04 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
21/12/01 13:56:04 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
21/12/01 13:57:57 WARN fs.LocalDirAllocator$AllocatorPerContext: Disk Error Exception: 
org.apache.hadoop.util.DiskChecker$DiskErrorException: Cannot create directory: /system/mapred/XX/.staging/_distcp-260350640
        at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:98)
        at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:77)
        at org.apache.hadoop.util.BasicDiskValidator.checkStatus(BasicDiskValidator.java:32)
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:367)
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:447)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:165)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
        at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:3549)
        at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:3343)
        at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:3319)
        at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2882)
        at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2921)
        at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:476)
        at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:450)
        at org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:155)
        at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)
        at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
        at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
        at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:368)
        at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
        at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:205)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:441)
21/12/01 13:57:57 ERROR tools.DistCp: Exception encountered 
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for hdfs://rbf-XXXXXX/system/mapred/XX/
.staging/_distcp-260350640/intermediate.1
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:463)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:165)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
        at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:3549)
        at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:3343)
        at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:3319)
        at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2882)
        at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2921)
        at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:476)
        at org.apache.hadoop.tools.util.DistCpUtils.sortListing(DistCpUtils.java:450)
        at org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:155)
        at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)
        at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
        at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
        at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:368)
        at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
        at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:205)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:441)

staging-dir:

<property>
  <name>yarn.app.mapreduce.am.staging-dir</name>
  <value>hdfs://rbf-XXXXXX/system/mapred</value>
</property>