[DRILL-7973] Fix GitHub CI intermittent failures

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.19.0
• Fix Version/s: 1.20.0
• Component/s: Tools, Build & Test
• Assignee: Vitalii Diravka
• Reporter: Vitalii Diravka
• Created: 15/Jul/21 20:41
• Updated: 22/Nov/21 19:45
• Resolved: 19/Oct/21 14:13

Description

The memory issue:
I got locally:

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.264 s - in org.apache.drill.exec.work.metadata.TestServerMetaProvider
[ERROR] Tests run: 19, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 7.713 s <<< FAILURE! - in org.apache.drill.exec.record.vector.TestValueVector
[ERROR] org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation  Time elapsed: 0.084 s  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<org.apache.drill.exec.exception.OversizedAllocationException> but was<org.apache.drill.exec.exception.OutOfMemoryException>
        at org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:107)
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2147483644 byte(s) of direct memory (used: 872415232, max: 2621440000)
        at org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:107)[ERROR] org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation  Time elapsed: 0.013 s  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<org.apache.drill.exec.exception.OversizedAllocationException> but was<org.apache.drill.exec.exception.OutOfMemoryException>
        at org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:165)
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2147483647 byte(s) of direct memory (used: 872415232, max: 2621440000)
        at org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:165)

for the command:

drill$ mvn install -V -ntp -DdirectMemoryMb=2500 -DmemoryMb=2000

and 

[INFO] Running org.apache.drill.exec.work.metadata.TestServerMetaProvider
[ERROR] Tests run: 19, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 6.358 s <<< FAILURE! - in org.apache.drill.exec.record.vector.TestValueVector
[ERROR] org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation  Time elapsed: 0.048 s  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<org.apache.drill.exec.exception.OversizedAllocationException> but was<org.apache.drill.exec.exception.OutOfMemoryException>
        at org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:107)
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2147483644 byte(s) of direct memory (used: 989855744, max: 2936012800)
        at org.apache.drill.exec.record.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:107)[ERROR] org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation  Time elapsed: 0.008 s  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<org.apache.drill.exec.exception.OversizedAllocationException> but was<org.apache.drill.exec.exception.OutOfMemoryException>
        at org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:165)
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2147483647 byte(s) of direct memory (used: 989855744, max: 2936012800)
        at org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:165)

for the command:

mvn install -V -ntp -DdirectMemoryMb=2800 -DmemoryMb=2000

There are also intermittent failures for several test cases, details in comments.

Issue Links

• fixes: DRILL-3192 TestDrillbitResilience#cancelWhenQueryIdArrives hangs
• fixes: DRILL-3167 When a query fails, Foreman should wait for all fragments to finish cleaning up before sending a FAILED state to the client
• fixes: DRILL-3193 TestDrillbitResilience#interruptingWhileFragmentIsBlockedInAcquiringSendingTicket hangs and fails
• fixes: DRILL-3194 TestDrillbitResilience#memoryLeaksWhenFailed hangs
• fixes: DRILL-3967 Broken Test: TestDrillbitResilience.cancelAfterEverythingIsCompleted()
• incorporates: DRILL-8030 Intermittent TestDrillbitResilience cancelInMiddleOfFetchingResults and foreman_runTryEnd failures
• relates to: DRILL-7844 Some GitHub Actions builds intermittently fail
• Testing discovered: DRILL-8012 Flaky tests TestExtendedTypes and TestNestedDateTimeTimestamp