Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12502

SetReplication OutOfMemoryError

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 3.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Setting the replication of a HDFS folder recursively can run out of memory. E.g. with a large /var/log directory:

      hdfs dfs -setrep -R -w 1 /var/log

      Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
      at java.util.Arrays.copyOfRange(Arrays.java:2694)
      at java.lang.String.<init>(String.java:203)
      at java.lang.String.substring(String.java:1913)
      at java.net.URI$Parser.substring(URI.java:2850)
      at java.net.URI$Parser.parse(URI.java:3046)
      at java.net.URI.<init>(URI.java:753)
      at org.apache.hadoop.fs.Path.initialize(Path.java:203)
      at org.apache.hadoop.fs.Path.<init>(Path.java:116)
      at org.apache.hadoop.fs.Path.<init>(Path.java:94)
      at org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:222)
      at org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:246)
      at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:689)
      at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
      at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
      at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
      at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
      at org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:268)
      at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347)
      at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308)
      at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347)
      at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308)
      at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347)
      at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308)
      at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347)
      at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308)
      at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:347)
      at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308)
      at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
      at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
      at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
      at org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:76)

        Attachments

        1. HADOOP-12502-10.patch
          12 kB
          Vinayakumar B
        2. HADOOP-12502-09.patch
          13 kB
          Vinayakumar B
        3. HADOOP-12502-08.patch
          12 kB
          Vinayakumar B
        4. HADOOP-12502-07.patch
          11 kB
          Wei-Chiu Chuang
        5. HADOOP-12502-06.patch
          15 kB
          Vinayakumar B
        6. HADOOP-12502-05.patch
          15 kB
          Vinayakumar B
        7. HADOOP-12502-04.patch
          17 kB
          Vinayakumar B
        8. HADOOP-12502-03.patch
          14 kB
          Vinayakumar B
        9. HADOOP-12502-02.patch
          12 kB
          Vinayakumar B
        10. HADOOP-12502-01.patch
          7 kB
          Vinayakumar B

          Issue Links

            Activity

              People

              • Assignee:
                vinayakumarb Vinayakumar B
                Reporter:
                philippsch Philipp Schuegerl
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: