[HBASE-12123] Failed assertion in BucketCache after HBASE-11331

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.98.7, 0.99.1
• Component/s: regionserver
• Assignee: Nick Dimiduk
• Reporter: Enis Soztutar
• Created: 30/Sep/14 19:34
• Updated: 06/Apr/18 17:54
• Resolved: 01/Oct/14 00:34

Description

As reported by enis

We have seen this in one of the test runs:

2014-09-26 05:31:19,788 WARN  [main-BucketCacheWriter-2] bucket.BucketCache: Failed doing drain
java.lang.AssertionError
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1239)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.doDrain(BucketCache.java:773)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:731)
  at java.lang.Thread.run(Thread.java:745)
2014-09-26 05:31:19,925 INFO  [main-BucketCacheWriter-2] bucket.BucketCache: main-BucketCacheWriter-2 exiting, cacheEnabled=true
2014-09-26 05:31:19,838 WARN  [main-BucketCacheWriter-1] bucket.BucketCache: Failed doing drain
java.lang.AssertionError
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1239)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.doDrain(BucketCache.java:773)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:731)
  at java.lang.Thread.run(Thread.java:745)
2014-09-26 05:31:19,791 WARN  [main-BucketCacheWriter-0] bucket.BucketCache: Failed doing drain
java.lang.AssertionError
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1239)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.doDrain(BucketCache.java:773)
  at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:731)
  at java.lang.Thread.run(Thread.java:745)
2014-09-26 05:31:19,926 INFO  [main-BucketCacheWriter-0] bucket.BucketCache: main-BucketCacheWriter-0 exiting, cacheEnabled=true
2014-09-26 05:31:19,926 INFO  [main-BucketCacheWriter-1] bucket.BucketCache: main-BucketCacheWriter-1 exiting, cacheEnabled=true

We are still running with assertions on in tests, and this block is failing the assertion. Seems important:

        if (data instanceof HFileBlock) {
          ByteBuffer sliceBuf = ((HFileBlock) data).getBufferReadOnlyWithHeader();
          sliceBuf.rewind();
          assert len == sliceBuf.limit() + HFileBlock.EXTRA_SERIALIZATION_SPACE;

Issue Links

• relates to: HBASE-11331 [blockcache] lazy block decompression