[IMPALA-5164] BenchmarkTest.Basic test is flaky

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.9.0
• Component/s: Infrastructure
• Labels: broken-build flaky
• Epic Color: ghx-label-7
• Assignee: Zach Amsden
• Reporter: Matthew Mulder
• Created: 04/Apr/17 21:57
• Updated: 22/May/18 18:28
• Resolved: 07/Jun/17 20:12

Description

The BenchmarkTest.Basic test simply compares how many times a function that memcopies 16 bytes of data can be executed within a give time versus a function that memcopies 128 bytes of data in the same amount of given time. Recently it had a string of failures, meaning that the function copying more data took less time. The StopWatch class is used to do the timing. I'm assuming this test is basically just a unit test for the StopWatch class. According to a comment in the StopWatch class, it is inaccurate if context switching occurs. So I think the test needs to call getrusage() before, in-between and after taking the two measurements and take the number of context switches during each measurement into account to determine if it is fair to compare the two measurements before deciding whether to fail. The test code is in be/src/util/benchmark-test.cc. The test failure looks like this.

11:07:28       Start 55: benchmark-test
11:07:29 55/88 Test #55: benchmark-test ...................***Failed    0.54 sec
11:07:29 Turning perftools heap leak checking off
11:07:29 [==========] Running 1 test from 1 test case.
11:07:29 [----------] Global test environment set-up.
11:07:29 [----------] 1 test from BenchmarkTest
11:07:29 [ RUN      ] BenchmarkTest.Basic
11:07:29 Rate 16 Byte: 48295
11:07:29 Rate 128 Byte: 72249.1
11:07:29 /home/jenkins/Impala/be/src/util/benchmark-test.cc:67: Failure
11:07:29 Expected: (rate_copy_128) < (rate_copy_16), actual: 72249.1 vs 48295
11:07:29 [  FAILED  ] BenchmarkTest.Basic (128 ms)
11:07:29 [----------] 1 test from BenchmarkTest (128 ms total)
11:07:29 
11:07:29 [----------] Global test environment tear-down
11:07:29 [==========] 1 test from 1 test case ran. (129 ms total)
11:07:29 [  PASSED  ] 0 tests.
11:07:29 [  FAILED  ] 1 test, listed below:
11:07:29 [  FAILED  ] BenchmarkTest.Basic
11:07:29 
11:07:29  1 FAILED TEST