[AURORA-1972] Flapping Excutor HealthChecker Test

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Executor
• Assignee: Unassigned
• Reporter: Stephan Erb
• Created: 09/Feb/18 14:25
• Updated: 09/Feb/18 14:25

Description

We currently have a set of flapping HealthChecking test that prevent our builds from passing (e.g. see https://reviews.apache.org/r/65565/).

                     ==================== FAILURES ====================
                      TestThreadedHealthCheckerWithDefaults.test_run_unhealthy_after_callback

                     self = <apache.aurora.executor.common.test_health_checker.TestThreadedHealthCheckerWithDefaults testMethod=test_run_unhealthy_after_callback>
                     mock_sleep = <MagicMock name='sleep' spec='builtin_function_or_method' id='4598488912'>

                         @mock.patch('apache.aurora.executor.common.health_checker.time.sleep', spec=time.sleep)
                         def test_run_unhealthy_after_callback(self, mock_sleep):
                           mock_sleep.return_value = None
                           health_status = [(True, None), (True, None), (False, 'failure-4'), (False, 'failure-5')]
                           self.health.side_effect = lambda: health_status.pop(0)
                           mock_is_set = mock.Mock(spec=threading._Event.is_set)
                           liveness = [False, False, False, False, True]
                           mock_is_set.side_effect = lambda: liveness.pop(0)
                           self.health_checker.threaded_health_checker.dead.is_set = mock_is_set
                           self.health_checker.threaded_health_checker.run()
                     >     assert mock_sleep.call_count == 4
                     E     AssertionError: assert 9403 == 4
                     E      +  where 9403 = <MagicMock name='sleep' spec='builtin_function_or_method' id='4598488912'>.call_count

                     .pants.d/pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/executor/common/test_health_checker.py:1292: AssertionError

Please notice the huge difference between actual and expected calls.

This works as expected:

./pants --cache-ignore --no-test-pytest-fast test.pytest  src/test/python::

This triggers the problem with traces as the one posted above:

./pants --cache-ignore --no-test-pytest-fast test.pytest  src/test/python::

The flapping seems to be dependent on how pants executes the tests. This seems to have a side effect on how time.sleep mocking is performed.

The relevant test class and the failing tests in particular are somewhat low-quality as they make heavy use of mocking.

How do we want to proceed here?