[HADOOP-1866] distcp requires large heapsize when copying many files

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.13.1
• Fix Version/s: 0.15.0
• Component/s: util
• Assignee: Christopher Douglas
• Reporter: Koji Noguchi
• Created: 10/Sep/07 16:37
• Updated: 24/Oct/07 21:42
• Resolved: 24/Oct/07 18:19

Description

Trying to distcp 1.5 million files with 1G client heapsize, failed with outofmemory.

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
exceeded
at java.util.regex.Pattern.compile(Pattern.java:1438)
at java.util.regex.Pattern.<init>(Pattern.java:1130)
at java.util.regex.Pattern.compile(Pattern.java:846)
at java.lang.String.replace(String.java:2208)
at org.apache.hadoop.fs.Path.normalizePath(Path.java:147)
at org.apache.hadoop.fs.Path.initialize(Path.java:137)
at org.apache.hadoop.fs.Path.<init>(Path.java:126)
at org.apache.hadoop.dfs.DfsPath.<init>(DfsPath.java:32)
at org.apache.hadoop.dfs.DistributedFileSystem$RawDistributedFileSystem.listPaths(DistributedFileSystem.java:214)
at org.apache.hadoop.fs.FileSystem.listPaths(FileSystem.java:483)
at org.apache.hadoop.fs.FileSystem.listPaths(FileSystem.java:496)
at org.apache.hadoop.fs.ChecksumFileSystem.listPaths(ChecksumFileSystem.java:539)
at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.setup(CopyFiles.java:327)
at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:762)
at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:808)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:818)

It would be nice if distcp doesn't require gigs of heapsize when copying large amount of files.