[HBASE-21323] Should not skip force updating for a sub procedure even if it has been finished

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0, 2.2.0, 2.1.1, 2.0.3
• Component/s: proc-v2
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 16/Oct/18 12:11
• Updated: 20/Oct/18 04:38
• Resolved: 19/Oct/18 22:33

Description

Keep seeing this

2018-10-16,20:03:02,027 WARN [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure WALs count=340 above the warning threshold 10. check running procedures to see if something is stuck.
2018-10-16,20:03:02,027 INFO [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=343
2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=994, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=995, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:02,870 WARN [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure WALs count=341 above the warning threshold 10. check running procedures to see if something is stuck.
2018-10-16,20:03:02,870 INFO [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=344
2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=994, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=995, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:03,816 WARN [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure WALs count=342 above the warning threshold 10. check running procedures to see if something is stuck.
2018-10-16,20:03:03,816 INFO [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=345
2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=994, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.
2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=995, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating.

Issue Links

• is related to: HBASE-21254 Need to find a way to limit the number of proc wal files