HBase
  1. HBase
  2. HBASE-3421

Very wide rows -- 30M plus -- cause us OOME

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.0
    • Fix Version/s: 0.90.5
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      A new config parameter, "hbase.hstore.compaction.kv.max", has been added to limit the number of rows processed in each iteration of the internal compaction code.
      Default value is 10.
      Show
      A new config parameter, "hbase.hstore.compaction.kv.max", has been added to limit the number of rows processed in each iteration of the internal compaction code. Default value is 10.

      Description

      From the list, see 'jvm oom' in http://mail-archives.apache.org/mod_mbox/hbase-user/201101.mbox/browser, it looks like wide rows – 30M or so – causes OOME during compaction. We should check it out. Can the scanner used during compactions use the 'limit' when nexting? If so, this should save our OOME'ing (or, we need to add to the next a max size rather than count of KVs).

      1. 3421.addendum
        0.7 kB
        Ted Yu
      2. HBASE-3421.patch
        0.8 kB
        Nate Putnam
      3. HBASE-34211-v2.patch
        1 kB
        Nate Putnam
      4. HBASE-34211-v3.patch
        2 kB
        Nate Putnam
      5. HBASE-34211-v4.patch
        1 kB
        Nate Putnam

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          Addendum fixes Store.FIXED_OVERHEAD

          Show
          Ted Yu added a comment - Addendum fixes Store.FIXED_OVERHEAD
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #6 (See https://builds.apache.org/job/HBase-0.92/6/)
          HBASE-3421 Very wide rows – 30M plus – cause us OOME (Nate Putnam)

          tedyu :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #6 (See https://builds.apache.org/job/HBase-0.92/6/ ) HBASE-3421 Very wide rows – 30M plus – cause us OOME (Nate Putnam) tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2239 (See https://builds.apache.org/job/HBase-TRUNK/2239/)
          HBASE-3421 Very wide rows – 30M plus – cause us OOME (Nate Putnam)

          tedyu :
          Files :

          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2239 (See https://builds.apache.org/job/HBase-TRUNK/2239/ ) HBASE-3421 Very wide rows – 30M plus – cause us OOME (Nate Putnam) tedyu : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Hide
          Ted Yu added a comment -

          Integrate to 0.90.5, 0.92 and TRUNK.

          Thanks for the patch Nate.

          Thanks for the review Michael.

          Show
          Ted Yu added a comment - Integrate to 0.90.5, 0.92 and TRUNK. Thanks for the patch Nate. Thanks for the review Michael.
          Hide
          stack added a comment -

          +1 on patch.

          Show
          stack added a comment - +1 on patch.
          Hide
          Ted Yu added a comment -

          +1 on patch v4.

          Show
          Ted Yu added a comment - +1 on patch v4.
          Hide
          Nate Putnam added a comment -

          Sorry about that. My mistake, I should be more careful when creating patches. v4 is the winner. Thanks for your patience.

          Show
          Nate Putnam added a comment - Sorry about that. My mistake, I should be more careful when creating patches. v4 is the winner. Thanks for your patience.
          Hide
          Ted Yu added a comment -

          Patch v3 included changes to HBaseConfiguration.java
          If I commit v3, Todd would kill me

          Show
          Ted Yu added a comment - Patch v3 included changes to HBaseConfiguration.java If I commit v3, Todd would kill me
          Hide
          Nate Putnam added a comment -

          Clarify code comment. Grant a license this time

          Show
          Nate Putnam added a comment - Clarify code comment. Grant a license this time
          Hide
          Nate Putnam added a comment -

          Clarify code comment.

          Show
          Nate Putnam added a comment - Clarify code comment.
          Hide
          Ted Yu added a comment -

          Patch v2 looks good.
          Minor comment: 10 should be replaced with new config below:

          +        // Limit this to 10 to avoid OOME
          

          The patch doesn't apply on TRUNK.
          Please prepare another patch for 0.92/TRUNK.

          Thanks

          Show
          Ted Yu added a comment - Patch v2 looks good. Minor comment: 10 should be replaced with new config below: + // Limit this to 10 to avoid OOME The patch doesn't apply on TRUNK. Please prepare another patch for 0.92/TRUNK. Thanks
          Hide
          Nate Putnam added a comment -

          Added hbase.hstore.compaction.kv.max config option.

          Show
          Nate Putnam added a comment - Added hbase.hstore.compaction.kv.max config option.
          Hide
          Nate Putnam added a comment -

          Sure. I'll submit a new patch tomorrow.

          Show
          Nate Putnam added a comment - Sure. I'll submit a new patch tomorrow.
          Hide
          Ted Yu added a comment -

          A limit of 10 is used here. Should a config parameter be introduced for the limit ?

          Show
          Ted Yu added a comment - A limit of 10 is used here. Should a config parameter be introduced for the limit ?
          Hide
          Nate Putnam added a comment -

          Been running in our prod cluster for a couple days without issue.

          Show
          Nate Putnam added a comment - Been running in our prod cluster for a couple days without issue.
          Hide
          stack added a comment -

          Anyone foresee an issue with just using a limit of 10 KVs on the scan(kvs) call in the compaction loop?

          No. Things might run imperceptibly slower.

          Show
          stack added a comment - Anyone foresee an issue with just using a limit of 10 KVs on the scan(kvs) call in the compaction loop? No. Things might run imperceptibly slower.
          Hide
          Todd Lipcon added a comment -

          Anyone foresee an issue with just using a limit of 10 KVs on the scan(kvs) call in the compaction loop? TestCompaction passes fine with that change.

          Show
          Todd Lipcon added a comment - Anyone foresee an issue with just using a limit of 10 KVs on the scan(kvs) call in the compaction loop? TestCompaction passes fine with that change.
          Hide
          Todd Lipcon added a comment -

          I think I'm seeing this on one workload as well - I added a "stats" command for the HFile tool, then ran it on a problematic 1.5GB gzip-compressed HFile that won't compact due to OOME.

          Key length: count: 272358037 min: 59 max: 2259 mean: 59.620136732003246
          Val length: count: 272358037 min: 0 max: 0 mean: 0.0
          Row size (bytes): count: 1287745 min: 67 max: 15615127736 mean: 14301.657317248368
          Row size (columns): count: 1287745 min: 1 max: 233061608 mean: 211.49997631518661

          so in this case there's actually a row with 233 million columns which takes up 15GB uncompressed!

          Show
          Todd Lipcon added a comment - I think I'm seeing this on one workload as well - I added a "stats" command for the HFile tool, then ran it on a problematic 1.5GB gzip-compressed HFile that won't compact due to OOME. Key length: count: 272358037 min: 59 max: 2259 mean: 59.620136732003246 Val length: count: 272358037 min: 0 max: 0 mean: 0.0 Row size (bytes): count: 1287745 min: 67 max: 15615127736 mean: 14301.657317248368 Row size (columns): count: 1287745 min: 1 max: 233061608 mean: 211.49997631518661 so in this case there's actually a row with 233 million columns which takes up 15GB uncompressed!
          Hide
          Nicolas Spiegelberg added a comment -

          For interested parties...

          From: Ted Yu
          Hi,
          I used the command you suggested in HBASE-3421 on a table and got:

          K: 0012F2157E58883070B9814047048E8B/v:_/1283909035492/Put/vlen=1308
          K: 0041A80A545C4CBF412865412065BF5E/v:_/1283909035492/Put/vlen=1311
          K: 00546F4AA313020E551E049E848949C6/v:_/1283909035492/Put/vlen=1866
          K: 0068CC263C81CE65B65FC5425EFEBBCD/v:_/1283909035492/Put/vlen=1191
          K: 006DB8745D6D1B624F77E0F06C177C0B/v:_/1283909035492/Put/vlen=1021
          K: 006F9037BD7A8F081B54C5B03756C143/v:_/1283909035492/Put/vlen=1382
          ...

          Can you briefly describe what conclusion can be drawn here ?

          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          From: Nicolas Spiegelberg

          You're basically seeing all the KeyValues in that HFile. The format is basically:

          K: <KeyValue.toString()>

          If you look at KeyValue.toString(), you'll see that the format is roughly:

          row/family:qualifier/timestamp/type/value_length

          So, it looks like you only have one qualifier per row and each row is roughly ~1500 bytes of data. For the user with the 30K columns per row, you should see an output that contains a ton of lines with the same row. If you grep that row, cut the number after vlen=, and sum the values, you can see the size of your rows on a per-Hfile basis.

          Show
          Nicolas Spiegelberg added a comment - For interested parties... From: Ted Yu Hi, I used the command you suggested in HBASE-3421 on a table and got: K: 0012F2157E58883070B9814047048E8B/v:_/1283909035492/Put/vlen=1308 K: 0041A80A545C4CBF412865412065BF5E/v:_/1283909035492/Put/vlen=1311 K: 00546F4AA313020E551E049E848949C6/v:_/1283909035492/Put/vlen=1866 K: 0068CC263C81CE65B65FC5425EFEBBCD/v:_/1283909035492/Put/vlen=1191 K: 006DB8745D6D1B624F77E0F06C177C0B/v:_/1283909035492/Put/vlen=1021 K: 006F9037BD7A8F081B54C5B03756C143/v:_/1283909035492/Put/vlen=1382 ... Can you briefly describe what conclusion can be drawn here ? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From: Nicolas Spiegelberg You're basically seeing all the KeyValues in that HFile. The format is basically: K: <KeyValue.toString()> If you look at KeyValue.toString(), you'll see that the format is roughly: row/family:qualifier/timestamp/type/value_length So, it looks like you only have one qualifier per row and each row is roughly ~1500 bytes of data. For the user with the 30K columns per row, you should see an output that contains a ton of lines with the same row. If you grep that row, cut the number after vlen=, and sum the values, you can see the size of your rows on a per-Hfile basis.
          Hide
          Nicolas Spiegelberg added a comment -

          Note that you can limit the number of StoreFiles that can be compacted at one time...

          Store.java#204: this.maxFilesToCompact =
          conf.getInt("hbase.hstore.compaction.max", 10)

          30M * 10 SF == 300MB. What is your RAM capacity? You are likely stuck on an merging outlier that exists in every SF. I would run:

          bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -f <FILE_NAME> -p |sed 's/V:.*$//g'|less

          on the HFiles in that Store to see what your high watermark is.

          Show
          Nicolas Spiegelberg added a comment - Note that you can limit the number of StoreFiles that can be compacted at one time... Store.java#204: this.maxFilesToCompact = conf.getInt("hbase.hstore.compaction.max", 10) 30M * 10 SF == 300MB. What is your RAM capacity? You are likely stuck on an merging outlier that exists in every SF. I would run: bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -f <FILE_NAME> -p |sed 's/V:.*$//g'|less on the HFiles in that Store to see what your high watermark is.

            People

            • Assignee:
              Nate Putnam
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development