[HBASE-13520] NullPointerException in TagRewriteCell

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0, 1.0.2, 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 21/Apr/15 02:43
• Updated: 04/Jul/15 12:29
• Resolved: 21/Apr/15 12:15

Description

Found via running IntegrationTestIngestWithVisibilityLabels with Kerberos enabled.

2015-04-20 18:54:36,712 ERROR [B.defaultRpcServer.handler=17,queue=2,port=16020] ipc.RpcServer: Unexpected throwable object
java.lang.NullPointerException
        at org.apache.hadoop.hbase.TagRewriteCell.getTagsLength(TagRewriteCell.java:157)
        at org.apache.hadoop.hbase.TagRewriteCell.heapSize(TagRewriteCell.java:186)
        at org.apache.hadoop.hbase.CellUtil.estimatedHeapSizeOf(CellUtil.java:568)
        at org.apache.hadoop.hbase.regionserver.DefaultMemStore.heapSizeChange(DefaultMemStore.java:1024)
        at org.apache.hadoop.hbase.regionserver.DefaultMemStore.internalAdd(DefaultMemStore.java:259)
        at org.apache.hadoop.hbase.regionserver.DefaultMemStore.upsert(DefaultMemStore.java:567)
        at org.apache.hadoop.hbase.regionserver.DefaultMemStore.upsert(DefaultMemStore.java:541)
        at org.apache.hadoop.hbase.regionserver.HStore.upsert(HStore.java:2154)
        at org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:7127)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:504)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2020)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31967)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2106)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$2.run(RpcExecutor.java:107)
        at java.lang.Thread.run(Thread.java:745)

HBASE-11870 tried to be tricky when only the tags of a Cell need to be altered in the write-pipeline by creating a TagRewriteCell which avoided copying all components of the original Cell. In an attempt to help free the tags on the old cell that we wouldn't be referencing anymore, TagRewriteCell nulls out the original byte[] tags.

This causes a problem in that the implementation of heapSize() as it getTagsLength() on the original Cell instead of the on this. Because the tags on the passed in Cell (which was also a TagRewriteCell) were null'ed out in the constructor, this results in a NPE by the byte array is null.

I believe this isn't observed in normal, unsecure deployments because there is only one RegionObserver/Coprocessor loaded that gets invoked via postMutationBeforeWAL. When there is only one RegionObserver, the TagRewriteCell isn't passed another TagRewriteCell, but instead a cell from the wire/protobuf. This means that the optimization isn't performed. When we have two (or more) observers that a TagRewriteCell passes through (and a new TagRewriteCell is created and the old TagRewriteCell's tags array is nulled), this enables the described-above NPE.

Issue Links

• is broken by: HBASE-11870 Optimization : Avoid copy of key and value for tags addition in AC and VC