[HBASE-21069] NPE in StoreScanner.updateReaders causes RS to crash

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.2
• Fix Version/s: 1.3.3, 1.4.7
• Hadoop Flags: Reviewed
• Assignee: Andrew Kyle Purtell
• Reporter: Thomas D'Silva
• Created: 17/Aug/18 21:50
• Updated: 09/Jul/19 06:40
• Resolved: 21/Aug/18 16:59

Description

I see the following NPE in the region server log for a table that is taking heavy writes.
I am not sure how the memStoreScanners variable gets set to null.

2018-08-17 19:59:23,682 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem - Committing store file ...
2018-08-17 19:59:23,684 INFO  [MemStoreFlusher.1] regionserver.HStore - Added hdfs://...., entries=919170, sequenceid=275114, filesize=22.6 M
2018-08-17 19:59:23,689 FATAL [MemStoreFlusher.1] regionserver.HRegionServer - ABORTING region server iotperf1dchbase1a-dnds22-2-prd.eng.sfdc.net,60020,1533915690501: Replay of WAL required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: ......
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2581)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2258)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2220)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2106)
        at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2031)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:508)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:478)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
        at java.util.ArrayList.<init>(ArrayList.java:177)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:827)
        at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1160)
        at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1133)
        at org.apache.hadoop.hbase.regionserver.HStore.access$900(HStore.java:120)
        at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2487)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2536)
        ... 9 more
2018-08-17 19:59:23,692 FATAL [MemStoreFlusher.1] regionserver.HRegionServer - RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.phoenix.coprocessor.ScanRegionObserver, org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver, org.apache.phoenix.hbase.index.Indexer, org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver, org.apache.hadoop.hbase.security.token.TokenProvider, org.apache.phoenix.coprocessor.ServerCachingEndpointImpl]

Issue Links

• duplicates: HBASE-21130 NullPointerException in StoreScanner
• is duplicated by: HBASE-22667 [Flush] NPE when region flushs
• relates to: HBASE-20322 CME in StoreScanner causes region server crash