Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3737

CompressedWritable throws OutOfMemoryError

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.18.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We were seeing OutOfMemoryErrors with stack traces like the following (Hadoop 0.17.0):

      java.lang.OutOfMemoryError
              at java.util.zip.Deflater.init(Native Method)
              at java.util.zip.Deflater.<init>(Deflater.java:123)
              at java.util.zip.Deflater.<init>(Deflater.java:132)
              at org.apache.hadoop.io.CompressedWritable.write(CompressedWritable.java:71)
              at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
              at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
              at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1016)
              [...]
      

      A Google search found the following long-standing issue in Java in which use of java.util.zip.Deflater causes an OutOfMemoryError:

      http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4797189

      CompressedWritable instantiates a Deflater, but does not call deflater.end(). It should do that in order to release the Deflater's resources immediately, instead of waiting for the object to be finalized.

      We applied this change locally and saw much improvement in the stability of memory usage of our app.

      This may also affect the SequenceFile compression types, because org.apache.hadoop.io.compress.zlib.BuiltInZlib

      {Deflater,Inflater} extend java.util.zip.{Deflater,Inflater}

      . org.apache.hadoop.io.compress.Compressor defines an end() method, but I do not see that this method is ever called.

        Attachments

        1. HADOOP-3737.patch
          0.5 kB
          Grant Glouser

          Activity

            People

            • Assignee:
              gglouser Grant Glouser
              Reporter:
              gglouser Grant Glouser
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: