Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-25698

Persistent IllegalReferenceCountException at scanner open when using TinyLfuBlockCache

    XMLWordPrintableJSON

    Details

      Description

      Persistent scanner open failure with offheap read path enabled.

      Not sure how it happened. Test scenario was HBase 1 cluster replicating to HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.

      Caused by: org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
              at org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
              at org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
              at org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
              at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
              at org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
              at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
              at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
              at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
              at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
              at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:249)
              at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
              at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
      

      Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:

      hbase:017:0> describe "IntegrationTestBigLinkedList"
      Table IntegrationTestBigLinkedList is ENABLED                                                                                               
      IntegrationTestBigLinkedList                                                                                                                
      COLUMN FAMILIES DESCRIPTION                                                                                                                 
      {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
      F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}     
      {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
      FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}    
      {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
      FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}    
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vjasani Viraj Jasani
                Reporter:
                apurtell Andrew Kyle Purtell
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: