HBase
  1. HBase
  2. HBASE-5387

Reuse compression streams in HFileBlock.Writer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/.

      1. 5387.txt
        12 kB
        Ted Yu
      2. ASF.LICENSE.NOT.GRANTED--D1719.1.patch
        10 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--D1719.2.patch
        12 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--D1719.3.patch
        16 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--D1719.4.patch
        12 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--D1719.5.patch
        12 kB
        Phabricator
      7. ASF.LICENSE.NOT.GRANTED--D1725.1.patch
        27 kB
        Phabricator
      8. ASF.LICENSE.NOT.GRANTED--D1725.2.patch
        15 kB
        Phabricator
      9. Fix-deflater-leak-2012-02-10_18_48_45.patch
        12 kB
        Mikhail Bautin
      10. Fix-deflater-leak-2012-02-11_17_13_10.patch
        13 kB
        Mikhail Bautin
      11. Fix-deflater-leak-2012-02-12_00_37_27.patch
        13 kB
        Mikhail Bautin

        Activity

        Mikhail Bautin created issue -
        Mikhail Bautin made changes -
        Field Original Value New Value
        Attachment Fix-deflater-leak-2012-02-10_18_48_45.patch [ 12514195 ]
        Ted Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ted Yu made changes -
        Assignee Mikhail Bautin [ mikhail ]
        Mikhail Bautin made changes -
        Description We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure any time. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/.
        We need to to reuse compression streams in HFileBlock.Writer instead of allocating them every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate a new GZIPOutputStream object and an associated native data structure every time we create a compression stream. The native data structure is only deallocated in the finalizer. This is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/.
        Ted Yu made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.94.0 [ 12316419 ]
        Affects Version/s 0.94.0 [ 12316419 ]
        Priority Major [ 3 ] Critical [ 2 ]
        Phabricator made changes -
        Attachment D1719.1.patch [ 12514205 ]
        Phabricator made changes -
        Attachment D1719.2.patch [ 12514243 ]
        Mikhail Bautin made changes -
        Phabricator made changes -
        Attachment D1719.3.patch [ 12514256 ]
        Phabricator made changes -
        Attachment D1719.4.patch [ 12514257 ]
        Phabricator made changes -
        Attachment D1719.5.patch [ 12514258 ]
        Ted Yu made changes -
        Attachment 5387.txt [ 12514260 ]
        Mikhail Bautin made changes -
        Mikhail Bautin made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Mikhail Bautin made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ted Yu made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Phabricator made changes -
        Attachment D1725.1.patch [ 12514422 ]
        Phabricator made changes -
        Attachment D1725.2.patch [ 12515506 ]
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Mikhail Bautin
            Reporter:
            Mikhail Bautin
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development