[HBASE-14702] TestZKProcedureControllers.testZKCoordinatorControllerWithSingleMemberCohort is a flakey

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0, 1.3.0, 2.0.0
• Component/s: test
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 27/Oct/15 03:44
• Updated: 23/Jun/22 19:07
• Resolved: 27/Oct/15 05:36

Description

Tests fail in TestZKProcedureControllers with some regularity. Here is an example: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.3/309/jdk=latest1.8,label=Hadoop/testReport/org.apache.hadoop.hbase.procedure/TestZKProcedureControllers/testZKCoordinatorControllerWithSingleMemberCohort/

org.mockito.exceptions.verification.TooManyActualInvocations: 
procedureMember.submitSubprocedure(<any>);
Wanted 1 time:
-> at org.apache.hadoop.hbase.procedure.TestZKProcedureControllers.verifyCohort(TestZKProcedureControllers.java:364)
But was 2 times. Undesired invocation:
-> at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.startNewSubprocedure(ZKProcedureMemberRpcs.java:214)

	at org.apache.hadoop.hbase.procedure.TestZKProcedureControllers.verifyCohort(TestZKProcedureControllers.java:364)
	at org.apache.hadoop.hbase.procedure.TestZKProcedureControllers.runMockCommitWithOrchestratedControllers(TestZKProcedureControllers.java:235)
	at org.apache.hadoop.hbase.procedure.TestZKProcedureControllers.testZKCoordinatorControllerWithSingleMemberCohort(TestZKProcedureControllers.java:158)

Tests in this suite fail always in this Mockito verify step. Always double the number of calls expected. Strikes me as a Mockito bug or Mockito usage interesting behavior. Unless someone intercedes, I'm just going to loosen this assert to be at least N rather than exactly N.

Lets see if it fails again soon... so I have another example to paste in here.