[DRILL-8030] Intermittent TestDrillbitResilience cancelInMiddleOfFetchingResults and foreman_runTryEnd failures

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.19.0
• Fix Version/s: 1.20.0
• Component/s: Tools, Build & Test
• Assignee: Vitalii Diravka
• Reporter: Vitalii Diravka
• Created: 03/Nov/21 07:34
• Updated: 22/Nov/21 18:36
• Resolved: 04/Nov/21 12:38

Description

DRILL-7908 fixes distributed deadlocks in TestDrillbitResilience and add better timing for simulation the different Drill states. But sometimes several tests failed.
1. Sometimes tests indicate memory leak:

Error:  Failures: 
3419Error:  org.apache.drill.exec.server.TestDrillbitResilience.cancelInMiddleOfFetchingResults
3420Error:    Run 1: TestDrillbitResilience.cancelInMiddleOfFetchingResults:375 We are leaking 3000000 bytes ==> expected: <0> but was: <3000000>

But actually there is no memory leak. Looks like Drill just check actual memory to early, when dot all fragments are closed, so adding timeout before final countAllocatedMemory fixes the issue.
The other reason of test failures - the queries were not in expected state before cancelling (for instance in STARTING state instead of RUNNING), so adding timeout before starting cancellation thread allows to wait the proper drill query state, which is expected to be for Drill  in test case before cancellation.
I don't have anymore test failures with NUM_RUNS = 1000 (@RepeatedTest) for the problematic test cases. 

2. The other test case which failed is:

Error:  Failures: 
3540Error:    TestDrillbitResilience.foreman_runTryEnd:289->testForeman:973->assertFailsWithException:960->assertFailsWithException:954 Query state should be FAILED (and not COMPLETED). ==> expected: <COMPLETED> but was: <FAILED>

It relates to DRILL-3167. The root cause here is the following: in some cases we are completing the query faster than run-try-end exception is injecetd and thrown in Foreman. The Completed state is acceptable for such cases

Issue Links

• fixes: DRILL-3192 TestDrillbitResilience#cancelWhenQueryIdArrives hangs
• fixes: DRILL-3052 canceling a fragment executor before it starts running will cause the Foreman to wait indefinitely for a terminal message from that fragment
• fixes: DRILL-3167 When a query fails, Foreman should wait for all fragments to finish cleaning up before sending a FAILED state to the client
• fixes: DRILL-3193 TestDrillbitResilience#interruptingWhileFragmentIsBlockedInAcquiringSendingTicket hangs and fails
• fixes: DRILL-3194 TestDrillbitResilience#memoryLeaksWhenFailed hangs
• fixes: DRILL-6228 Random failures of TestDrillbitResilience tests
• fixes: DRILL-3967 Broken Test: TestDrillbitResilience.cancelAfterEverythingIsCompleted()
• is part of: DRILL-7973 Fix GitHub CI intermittent failures
• is related to: DRILL-3163 Fix hang/ leak issue exposed by TestDrillbitResilience#foreman_runTryEnd