[AURORA-836] test_integration_success in test_thermos_task_runner is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: Technical Debt, Thermos
• Sprint: Aurora Q4 Sprint 1
• Assignee: Bill Farner
• Reporter: Bill Farner
• Created: 14/Oct/14 22:39
• Updated: 28/Oct/14 22:38
• Resolved: 14/Oct/14 22:59

Description

This fails consistently on my workstation, but seems to pass on mac laptops and jenkins.

$ ./pants build --timeout=60 src/test/python/apache/aurora/executor:thermos_task_runner -vxs
Build operating on top level addresses: set([BuildFileAddress(/home/wfarner/code/aurora/src/test/python/apache/aurora/executor/BUILD, thermos_task_runner)])
======================================================================== test session starts =========================================================================
platform linux2 -- Python 2.7.6 -- py-1.4.25 -- pytest-2.6.3 -- /usr/bin/python2.7
plugins: cov, timeout
collected 8 items 

src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_success Writing log files to disk in /tmp/tmpX_YDxu
Build operating on top level addresses: set([BuildFileAddress(/home/wfarner/code/aurora/src/main/python/apache/aurora/executor/bin/BUILD, thermos_runner)])
Building PythonBinary PythonBinary(BuildFileAddress(/home/wfarner/code/aurora/src/main/python/apache/aurora/executor/bin/BUILD, thermos_runner)):
Wrote /home/wfarner/code/aurora/dist/thermos_runner.pex
Writing log files to disk in /tmp/tmpX_YDxu
FAILED

============================================================================== FAILURES ==============================================================================
_____________________________________________________ TestThermosTaskRunnerIntegration.test_integration_success ______________________________________________________

self = <test_thermos_task_runner.TestThermosTaskRunnerIntegration object at 0x7f89258d9090>

    def test_integration_success(self):
      with self.yield_sleepy(ThermosTaskRunner, sleep=0, exit_code=0) as task_runner:
>       task_runner.start()

src/test/python/apache/aurora/executor/test_thermos_task_runner.py:136: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/tmp/tmpxQyM2Q/apache/aurora/executor/thermos_task_runner.py:282: in start
    self.wait_start(timeout=timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <apache.aurora.executor.thermos_task_runner.ThermosTaskRunner object at 0x7f89258eaa90>, timeout = Amount(1, mins)

    def wait_start(self, timeout=MAX_WAIT):
      log.debug('Waiting for task to start.')
    
      def is_started():
        return self._monitor and (self._monitor.active or self._monitor.finished)
    
      waited = Amount(0, Time.SECONDS)
    
      while waited < timeout:
        if not is_started():
          log.debug('  - sleeping...')
          self._clock.sleep(self.POLL_INTERVAL.as_(Time.SECONDS))
          waited += self.POLL_INTERVAL
        else:
          break
    
        if not self.is_alive:
          if self._popen_rc != 0:
            raise TaskError('Task failed: %s' % self._popen_reason())
          else:
>           log.info('Task runner exited: %s' % self._popen_reason())
E           AttributeError: 'ThermosTaskRunner' object has no attribute '_popen_reason'

/tmp/tmpxQyM2Q/apache/aurora/executor/thermos_task_runner.py:304: AttributeError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================== 1 failed in 2.42 seconds ======================================================================
src.test.python.apache.aurora.executor.thermos_task_runner                      .....   FAILURE