2019-02-06 07:40:04,653 INFO [PEWorker-12] assignment.TransitRegionStateProcedure: Starting pid=44148, ppid=44140, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=bachehra_dummy-table, region=7def9c5afad4e3ca30df4535b5ad4776, ASSIGN; rit=OPEN, location=bn01ap6376e6dd6,16020,1549461085241; forceNewPlan=false, retain=true 2019-02-06 07:40:04,819 INFO [PEWorker-9] assignment.RegionStateStore: pid=44148 updating hbase:meta row=7def9c5afad4e3ca30df4535b5ad4776, regionState=OPENING, regionLocation=********************,16020,1549467393277 2019-02-06 07:58:18,140 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-9(pid=44148), run time 18mins, 13.321sec 2019-02-06 07:58:36,321 WARN [PEWorker-9] assignment.TransitRegionStateProcedure: Failed transition, suspend 1secs pid=44148, ppid=44140, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN, hasLock=true; TransitRegionStateProcedure table=bachehra_dummy-table, region=7def9c5afad4e3ca30df4535b5ad4776, ASSIGN; rit=OPEN, location=bn01ap4c3d3316e,16020,1549467393277; waiting on rectified condition fixed by other Procedure or operator intervention org.apache.hadoop.hbase.DoNotRetryIOException: hconnection-0x310dde0 closed at org.apache.hadoop.hbase.client.ConnectionImplementation.checkClosed(ConnectionImplementation.java:583) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:764) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ConnectionUtils.java:131) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:758) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ConnectionUtils.java:131) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:729) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ConnectionUtils.java:131) at org.apache.hadoop.hbase.client.ConnectionImplementation.getRegionLocation(ConnectionImplementation.java:611) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.getRegionLocation(ConnectionUtils.java:131) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:72) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:223) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:542) at org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateRegionLocation(RegionStateStore.java:207) at org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateUserRegionLocation(RegionStateStore.java:201) at org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateRegionLocation(RegionStateStore.java:139) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.transitStateAndUpdate(AssignmentManager.java:1467) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.regionOpening(AssignmentManager.java:1482) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.openRegion(TransitRegionStateProcedure.java:183) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:318) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:102) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:189) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:290) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:102) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:959) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1645) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1961) 2019-02-06 07:58:36,321 INFO [PEWorker-9] procedure2.TimeoutExecutorThread: ADDED pid=44148, ppid=44140, state=WAITING_TIMEOUT:REGION_STATE_TRANSITION_OPEN, hasLock=true; TransitRegionStateProcedure table=bachehra_dummy-table, region=7def9c5afad4e3ca30df4535b5ad4776, ASSIGN; timeout=1003, timestamp=1549468717324 2019-02-06 07:59:16,925 ERROR [RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16000] ipc.RpcServer: Unexpected throwable object java.lang.NullPointerException at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportTransition(AssignmentManager.java:950) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:922) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:838) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:883) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1584) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15176) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2019-02-06 07:59:18,714 INFO [main] master.HMaster: STARTING service HMaster 2019-02-06 07:59:23,141 INFO [master/**********************:16000:becomeActiveMaster] master.ActiveMasterManager: Another master is the active master, **************,16000,1549465516862; waiting to become the next active master 2019-02-06 09:00:39,751 INFO [master/**********************:16000:becomeActiveMaster] master.ActiveMasterManager: Registered as active master=***************,16000,1549468758937 2019-02-06 09:00:47,168 WARN [master/**********************:16000:becomeActiveMaster] wal.WALProcedureStore: Unable to read tracker for hdfs://******************************/wdphbase/bachehra_Hbase-ingestion-test_30RS_300R-100k-compaction-2/MasterProcWALs/pv2-00000000000000000117.log org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat$InvalidWALDataException: Invalid Trailer version. got 194 expected 1 at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.readTrailer(ProcedureWALFormat.java:186) at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.readTrailer(ProcedureWALFile.java:91) at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.readTracker(ProcedureWALFile.java:98) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLog(WALProcedureStore.java:1385) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:1334) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:417) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:582) at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1550) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:954) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2355) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:592) at java.lang.Thread.run(Thread.java:745) 2019-02-06 09:00:47,186 INFO [master/**********************:16000:becomeActiveMaster] wal.ProcedureWALFormatReader: Rebuilding tracker for hdfs://**************************/wdphbase/bachehra_Hbase-ingestion-test_30RS_300R-100k-compaction-2/MasterProcWALs/pv2-00000000000000000117.log 2019-02-06 09:00:47,229 WARN [master/**********************:16000:becomeActiveMaster] wal.ProcedureWALFormatReader: Nothing left to decode. Exiting with missing EOF, log=hdfs://*********************/wdphbase/bachehra_Hbase-ingestion-test_30RS_300R-100k-compaction-2/MasterProcWALs/pv2-00000000000000000117.log 2019-02-06 09:00:47,229 INFO [master/**********************:16000:becomeActiveMaster] wal.ProcedureWALFormatReader: Read 8 entries in hdfs://*****************/wdphbase/bachehra_Hbase-ingestion-test_30RS_300R-100k-compaction-2/MasterProcWALs/pv2-00000000000000000117.log 2019-02-06 09:00:48,012 ERROR [master/**********************:16000:becomeActiveMaster] wal.WALProcedureTree: Orphan procedure: Procedure(pid=44148, ppid=44140, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure)