[HBASE-21130] NullPointerException in StoreScanner

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.3.1
• Component/s: regionserver
• Assignee: Unassigned
• Reporter: Chandra Sekhar
• Created: 29/Aug/18 10:39
• Updated: 29/Aug/18 21:16
• Resolved: 29/Aug/18 21:16

Description

I've created a script that will put continuous put one record (size 2.5KB) and flush immediately – in middle am doing compaction at regular intervals. Rate of flushes are around 20flushes/sec. After some time, my RS aborted and never came up back
with the following error

2018-08-29 11:34:34,183 DEBUG [flush-table-TestTable_client_1258196-thread-1] regionserver.RSRpcServices: Closing region operation on TestTable_client_1,32816,1535513244999.762a3e633b03e5f847f357aca28768d0.
2018-08-29 11:34:34,183 INFO  [RpcServer.FifoWFPBQ.default.handler=49,queue=4,port=16040] regionserver.RSRpcServices: flush table task succeed 1, failed 10.
2018-08-29 11:34:34,280 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=230, memsize=4.2 K, hasBloomFilter=false, into tmp file hdfs://hacluster/hbase/data/hbase/meta/1588230740/.tmp/1cf1deee293848b0bea08940696dbd2a
2018-08-29 11:34:34,290 INFO  [MemStoreFlusher.0] regionserver.StoreFile$Reader: Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1cf1deee293848b0bea08940696dbd2a
2018-08-29 11:34:34,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://hacluster/hbase/data/hbase/meta/1588230740/.tmp/1cf1deee293848b0bea08940696dbd2a as hdfs://hacluster/hbase/data/hbase/meta/1588230740/info/1cf1deee293848b0bea08940696dbd2a
2018-08-29 11:34:34,304 INFO  [MemStoreFlusher.0] regionserver.StoreFile$Reader: Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1cf1deee293848b0bea08940696dbd2a
2018-08-29 11:34:34,304 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://hacluster/hbase/data/hbase/meta/1588230740/info/1cf1deee293848b0bea08940696dbd2a, entries=13, sequenceid=230, filesize=6.6 K
2018-08-29 11:34:34,307 FATAL [MemStoreFlusher.0] regionserver.HRegionServer: ABORTING region server host-xxxx,16040,1535454741321: Replay of WAL required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: hbase:meta,,1
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2578)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2255)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2217)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2108)
        at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2034)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:505)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:475)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:263)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
        at java.util.ArrayList.<init>(ArrayList.java:177)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:826)
        at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1117)
        at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1090)
        at org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:120)
        at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2450)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2533)
        ... 9 more
2018-08-29 11:34:34,307 FATAL [MemStoreFlusher.0] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.

Issue Links

• is duplicated by: HBASE-21069 NPE in StoreScanner.updateReaders causes RS to crash