Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-3421

Very wide rows -- 30M plus -- cause us OOME

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.90.0
    • 0.90.5
    • None
    • None
    • Reviewed
    • Hide
      A new config parameter, "hbase.hstore.compaction.kv.max", has been added to limit the number of rows processed in each iteration of the internal compaction code.
      Default value is 10.
      Show
      A new config parameter, "hbase.hstore.compaction.kv.max", has been added to limit the number of rows processed in each iteration of the internal compaction code. Default value is 10.

    Description

      From the list, see 'jvm oom' in http://mail-archives.apache.org/mod_mbox/hbase-user/201101.mbox/browser, it looks like wide rows – 30M or so – causes OOME during compaction. We should check it out. Can the scanner used during compactions use the 'limit' when nexting? If so, this should save our OOME'ing (or, we need to add to the next a max size rather than count of KVs).

      Attachments

        1. HBASE-3421.patch
          0.8 kB
          Nate Putnam
        2. HBASE-34211-v2.patch
          1 kB
          Nate Putnam
        3. HBASE-34211-v3.patch
          2 kB
          Nate Putnam
        4. HBASE-34211-v4.patch
          1 kB
          Nate Putnam
        5. 3421.addendum
          0.7 kB
          Ted Yu

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. HBASE-3929 Add option to HFile tool to produce basic stats

          • Major - Major loss of function.
          • Closed

          Activity

            People

              nputnam Nate Putnam
              stack Michael Stack
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: