[HBASE-14632] Region server aborts due to unguarded dereference of Reader

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0, 1.3.0, 1.1.3, 2.0.0
• Component/s: regionserver
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 16/Oct/15 22:32
• Updated: 27/Jan/16 15:28
• Resolved: 06/Nov/15 20:08

Description

I noticed the following in one run of org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster-output.txt :

2015-10-16 09:46:33,108 INFO  [main] client.HBaseAdmin$10(1233): Started disable of testStoreFileReferenceCreationWhenSplitPolicySaysToSkipRangeCheck
2015-10-16 09:46:33,108 INFO  [B.defaultRpcServer.handler=4,queue=0,port=38813] master.HMaster(1908): Client=hbase/null disable                                                   testStoreFileReferenceCreationWhenSplitPolicySaysToSkipRangeCheck
2015-10-16 09:46:33,223 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=38813] procedure2.ProcedureExecutor(654): Procedure DisableTableProcedure                                (table=testStoreFileReferenceCreationWhenSplitPolicySaysToSkipRangeCheck) id=24 owner=hbase state=RUNNABLE:DISABLE_TABLE_PREPARE added to the store.
2015-10-16 09:46:33,225 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=38813] master.MasterRpcServices(1057): Checking to see if procedure is done procId=24
2015-10-16 09:46:33,230 DEBUG [ProcedureExecutor-22] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/                                                     testStoreFileReferenceCreationWhenSplitPolicySaysToSkipRangeCheck/write-master:388130000000001
2015-10-16 09:46:33,320 WARN  [RS:0;cn012:53683] regionserver.HStore(1910): StoreFile hdfs://localhost:40022/user/hbase/test-data/f09d7163-94f7-4218-b1b0-43dfc733a37b/data/      default/testStoreFileReferenceCreationWhenSplitPolicySaysToSkipRangeCheck/dc90661cebac678ac508ed98093fc3e9/f/fffae6d1a0234c1791d8098cbcdb2c5e has a null Reader
2015-10-16 09:46:33,320 WARN  [RS:0;cn012:53683] regionserver.HStore(1924): StoreFile hdfs://localhost:40022/user/hbase/test-data/f09d7163-94f7-4218-b1b0-43dfc733a37b/data/      default/testStoreFileReferenceCreationWhenSplitPolicySaysToSkipRangeCheck/dc90661cebac678ac508ed98093fc3e9/f/fffae6d1a0234c1791d8098cbcdb2c5e has a null Reader
2015-10-16 09:46:33,321 WARN  [RS:0;cn012:53683] regionserver.HStore(1924): StoreFile hdfs://localhost:40022/user/hbase/test-data/f09d7163-94f7-4218-b1b0-43dfc733a37b/data/      default/testStoreFileReferenceCreationWhenSplitPolicySaysToSkipRangeCheck/dc90661cebac678ac508ed98093fc3e9/f/fffae6d1a0234c1791d8098cbcdb2c5e has a null Reader
2015-10-16 09:46:33,321 FATAL [RS:0;cn012:53683] regionserver.HRegionServer(2078): ABORTING region server cn012.l42scl.hortonworks.com,53683,1445013948320: Unhandled: null
java.lang.NullPointerException
  at org.apache.hadoop.hbase.regionserver.HStore.getTotalStaticIndexSize(HStore.java:1936)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:1470)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1206)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1149)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:965)
  at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
  at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108)
  at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:356)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
  at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:302)
  at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138)
  at java.lang.Thread.run(Thread.java:745)

Here is related code:

  public long getTotalStaticIndexSize() {
    long size = 0;
    for (StoreFile s : this.storeEngine.getStoreFileManager().getStorefiles()) {
      size += s.getReader().getUncompressedDataIndexSize();
    }
    return size;
  }

Some methods, such as getStorefilesIndexSize(), guard against null Reader by checking r against null.

Same practice should be followed in other getters.