[HBASE-17020] keylen in midkey() dont computed correctly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4, 2.0.0
• Fix Version/s: 1.4.0, 1.2.5, 0.98.24, 1.1.8, 2.0.0
• Component/s: HFile
• Hadoop Flags: Reviewed
• Assignee: Yu Sun
• Reporter: Yu Sun
• Created: 04/Nov/16 11:06
• Updated: 21/Mar/17 16:51
• Resolved: 11/Nov/16 02:46

Description

in CellBasedKeyBlockIndexReader.midkey():

          ByteBuff b = midLeafBlock.getBufferWithoutHeader();
          int numDataBlocks = b.getIntAfterPosition(0);
          int keyRelOffset = b.getIntAfterPosition(Bytes.SIZEOF_INT * (midKeyEntry + 1));
          int keyLen = b.getIntAfterPosition(Bytes.SIZEOF_INT * (midKeyEntry + 2)) - keyRelOffset;

the local varible keyLen get this should be total length of: SECONDARY_INDEX_ENTRY_OVERHEAD + firstKey.length;

the code is:

    void add(byte[] firstKey, long blockOffset, int onDiskDataSize,
        long curTotalNumSubEntries) {
      // Record the offset for the secondary index
      secondaryIndexOffsetMarks.add(curTotalNonRootEntrySize);
      curTotalNonRootEntrySize += SECONDARY_INDEX_ENTRY_OVERHEAD
          + firstKey.length;

when the midkey last entry of a leaf-level index block, this may throw:

2016-10-01 12:27:55,186 ERROR [MemStoreFlusher.0] regionserver.MemStoreFlusher: Cache flusher failed for entry [flush region pora_6_item_feature,0061:,1473838922457.12617bc4ebbfd171018bf96ac9bdd2a7.]
java.lang.ArrayIndexOutOfBoundsException
at org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray(ByteBufferUtils.java:936)
at org.apache.hadoop.hbase.nio.SingleByteBuff.toBytes(SingleByteBuff.java:303)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.midkey(HFileBlockIndex.java:419)
at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.midkey(HFileReaderImpl.java:1519)
at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1520)
at org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:706)
at org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1983)
at org.apache.hadoop.hbase.regionserver.ConstantFamilySizeRegionSplitPolicy.getSplitPoint(ConstantFamilySizeRegionSplitPolicy.java:77)
at org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7756)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:513)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
at java.lang.Thread.run(Thread.java:756)

Issue Links

• is duplicated by: HBASE-17113 finding middle key in HFileV2 is always wrong and can cause IndexOutOfBoundsException
• is related to: HBASE-17070 backport HBASE-17020 to 1.3.1