[HBASE-20131] NPE in MoveRegionProcedure via IntegrationTestLoadAndVerify with CM

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Implemented
• Fix Version/s: 2.0.0
• Component/s: proc-v2
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 04/Mar/18 15:34
• Updated: 14/Mar/18 19:45
• Resolved: 13/Mar/18 19:38

Description

I believe the error is that a MoveRegionProcedure comes in via ChaosMonkey for an unassigned region that was from a disabled table (also due to CM) which causes an NPE as we try to set a null original location into the protobuf which fails.

2018-03-02 23:07:00,146 ERROR [RpcServer.default.FPBQ.Fifo.handler=23,queue=2,port=20000] ipc.RpcServer: Unexpected throwable object 
java.lang.NullPointerException
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos$MoveRegionStateData$Builder.setSourceServer(MasterProcedureProtos.java:26127)
	at org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.serializeStateData(MoveRegionProcedure.java:133)
	at org.apache.hadoop.hbase.procedure2.ProcedureUtil.convertToProtoProcedure(ProcedureUtil.java:198)
	at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.writeEntry(ProcedureWALFormat.java:211)
	at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.writeInsert(ProcedureWALFormat.java:222)
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.insert(WALProcedureStore.java:490)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:863)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:832)
	at org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.submitProcedure(ProcedureSyncWait.java:111)
	at org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:561)
	at org.apache.hadoop.hbase.master.HMaster.move(HMaster.java:1707)
	at org.apache.hadoop.hbase.master.MasterRpcServices.moveRegion(MasterRpcServices.java:1324)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

IntegrationTestLoadAndVerify also failed, but I'm not sure if it's related to this, or just a problem with the test. The test failed because the table was left offline after it was disabled, and appears to not have been re-enabled. Still debugging that side..