Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0-beta-1, 2.7.0, 2.6.1, 2.5.10
-
Reviewed
Description
For MOB table, RSMobFileCleanerChore is responsible for cleaning MOB files that are no longer referebced by the region located in the current RegionServer.
RSMobFileCleanerChore get the the information of MOB files by reading the storefile, as fallow.
```java
// RSMobFileCleanerChore.chore()
sf.initReader();
byte[] mobRefData = sf.getMetadataValue(HStoreFile.MOB_FILE_REFS);
byte[] bulkloadMarkerData = sf.getMetadataValue(HStoreFile.BULKLOAD_TASK_KEY);
// close store file to avoid memory leaks
sf.closeStoreFile(true);
```
There is an issue in here, if the StoreFileReader was not created by RSMobFileCleanerChore, but RSMobFileCleanerChore closed it, which will cause the thread that created the object to be unusable, resuting ERROR finally.
Reproduction:
This is an occasional problem, but the probability of its occurrence can be increased by making the following modifications.
1. Setting hbase.master.mob.cleaner.period from 24h to 10s, and restart hbase.
2. Puting some mob data into a MOB table.
3. At the same time, executing compaction command for the MOB table, and it is possible that this problem may occur.
The error logs as follow.
ERROR: java.io.IOException: Cannot invoke "org.apache.hadoop.hbase.regionserver.StoreFileReader.getMaxTimestamp()" because the return value of "org.apache.hadoop.hbase.regionserver.HStoreFile.getReader()" is null
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:512)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hbase.regionserver.StoreFileReader.getMaxTimestamp()" because the return value of "org.apache.hadoop.hbase.regionserver.HStoreFile.getReader()" is null
at org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.lambda$getUnneededFiles$3(DefaultStoreFileManager.java:235)