[HBASE-26142] NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.4.0
• Fix Version/s: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
• Hadoop Flags: Reviewed
• Assignee: chenglei
• Reporter: chenglei
• Created: 27/Jul/21 08:09
• Updated: 08/Aug/21 03:41
• Resolved: 05/Aug/21 13:33

Description

The default value of hbase.hregion.memstore.mslab.indexchunksize.percent introduced by HBASE-24892 is 0.1, but when we use DefaultMemStore by default , which has no IndexChunk and ChunkCreator.indexChunksPool is useless(IndexChunk is only used by CompactingMemStore), so we set hbase.hregion.memstore.mslab.indexchunksize.percent to 0 to save memory space,
But after running a while, the RegionServer throws NullPointerException and abort:

   Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310)
        at org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608)
        at org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297)
        at org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268)
        at org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149)
        at org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251)
        at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244)
        at org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137)
        at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963)

The problem is caused by line 608 in ChunkCreator.putbackChunks : Chunk.isIndexChunk incorrectly returns true for DataChunk and
unexpectedly invokes indexChunksPool.putbackChunks， while indexChunksPool is null:

594 synchronized void putbackChunks(Set<Integer> chunks) {
595    // if there is no pool just try to clear the chunkIdMap in case there is something
596    if (dataChunksPool == null && indexChunksPool == null) {
597      this.removeChunks(chunks);
598      return;
599    }
600
601   // if there is a pool, go over all chunk IDs that came back, the chunks may be from pool or not
602    for (int chunkID : chunks) {
603     // translate chunk ID to chunk, if chunk initially wasn't in pool
604      // this translation will (most likely) return null
605      Chunk chunk = ChunkCreator.this.getChunk(chunkID);
606     if (chunk != null) {
607        if (chunk.isFromPool() && chunk.isIndexChunk()) {
608          indexChunksPool.putbackChunks(chunk);

For DataChunk , Chunk.isIndexChunk return true because Chunk.isIndexChunk determines the type of chunk based on Chunk.size

 boolean isIndexChunk() {
    return size == ChunkCreator.getInstance().getChunkSize(ChunkCreator.ChunkType.INDEX_CHUNK);
  }

and ChunkCreator.getChunkSize incorrectly return DataChunk size when ChunkCreator.indexChunksPool is null:

    int getChunkSize(ChunkType chunkType) {
    switch (chunkType) {
      case INDEX_CHUNK:
        if (indexChunksPool != null) {
          return indexChunksPool.getChunkSize();
        }
      case DATA_CHUNK:
        if (dataChunksPool != null) {
          return dataChunksPool.getChunkSize();
        } else { // When pools are empty
          return chunkSize;
        }
      default:
        throw new IllegalArgumentException(
                "chunkType must either be INDEX_CHUNK or DATA_CHUNK");
    }
  }

In my opinion, in addition to erroneous implementation of ChunkCreator.getChunkSize, we would better not determine the type of Chunk based on Chunk.size, because
hbase.hregion.memstore.mslab.indexchunksize.percent is set by user and the size of IndexChunk and DataChunk could be the same.Tagged a ChunkType to Chunk is a better choice.

Issue Links

• is depended upon by: HBASE-26143 The default value of 'hbase.hregion.memstore.mslab.indexchunksize.percent' should be 0
• relates to: HBASE-24892 config 'hbase.hregion.memstore.mslab.indexchunksize' not be used
• links to: GitHub Pull Request #3531