[KUDU-2631] DeleteTableITest.TestNoDeleteTombstonedTablets is a little flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.1
• Fix Version/s: 1.8.0
• Assignee: Alexey Serbin
• Reporter: William Berkeley
• Created: 03/Dec/18 23:03
• Updated: 04/Dec/18 00:04
• Resolved: 04/Dec/18 00:04

Description

delete_table-itest's TestNoDeleteTombstonedTablets test case is a little flaky since 87dcaf34d3. which implemented removal of excess replicas in the master. The test does the following:

A. Find the leader L of tablet T
B. Find a server X that's not the leader L but has a replica of T.
C. Find a server Y that has no replica of T.
D. Add a replica of T to Y (as a VOTER).
E. Remove the replica of T from X.
F. Wait and check that the master doesn't delete the tombstoned replica of T on Y, even after restart.

The problem is that step D makes T over-replicated, so E is racing with the master removing Y. Rarely, the master wins the race and its config change is pending when step E occurs, which causes E to fail because a tablet will only accept one config change at a time.