Uploaded image for project: 'Aurora'
  1. Aurora
  2. AURORA-836

test_integration_success in test_thermos_task_runner is flaky

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Technical Debt, Thermos
    • Labels:
      None
    • Sprint:
      Aurora Q4 Sprint 1

      Description

      This fails consistently on my workstation, but seems to pass on mac laptops and jenkins.

      $ ./pants build --timeout=60 src/test/python/apache/aurora/executor:thermos_task_runner -vxs
      Build operating on top level addresses: set([BuildFileAddress(/home/wfarner/code/aurora/src/test/python/apache/aurora/executor/BUILD, thermos_task_runner)])
      ======================================================================== test session starts =========================================================================
      platform linux2 -- Python 2.7.6 -- py-1.4.25 -- pytest-2.6.3 -- /usr/bin/python2.7
      plugins: cov, timeout
      collected 8 items 
      
      src/test/python/apache/aurora/executor/test_thermos_task_runner.py::TestThermosTaskRunnerIntegration::test_integration_success Writing log files to disk in /tmp/tmpX_YDxu
      Build operating on top level addresses: set([BuildFileAddress(/home/wfarner/code/aurora/src/main/python/apache/aurora/executor/bin/BUILD, thermos_runner)])
      Building PythonBinary PythonBinary(BuildFileAddress(/home/wfarner/code/aurora/src/main/python/apache/aurora/executor/bin/BUILD, thermos_runner)):
      Wrote /home/wfarner/code/aurora/dist/thermos_runner.pex
      Writing log files to disk in /tmp/tmpX_YDxu
      FAILED
      
      ============================================================================== FAILURES ==============================================================================
      _____________________________________________________ TestThermosTaskRunnerIntegration.test_integration_success ______________________________________________________
      
      self = <test_thermos_task_runner.TestThermosTaskRunnerIntegration object at 0x7f89258d9090>
      
          def test_integration_success(self):
            with self.yield_sleepy(ThermosTaskRunner, sleep=0, exit_code=0) as task_runner:
      >       task_runner.start()
      
      src/test/python/apache/aurora/executor/test_thermos_task_runner.py:136: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      /tmp/tmpxQyM2Q/apache/aurora/executor/thermos_task_runner.py:282: in start
          self.wait_start(timeout=timeout)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <apache.aurora.executor.thermos_task_runner.ThermosTaskRunner object at 0x7f89258eaa90>, timeout = Amount(1, mins)
      
          def wait_start(self, timeout=MAX_WAIT):
            log.debug('Waiting for task to start.')
          
            def is_started():
              return self._monitor and (self._monitor.active or self._monitor.finished)
          
            waited = Amount(0, Time.SECONDS)
          
            while waited < timeout:
              if not is_started():
                log.debug('  - sleeping...')
                self._clock.sleep(self.POLL_INTERVAL.as_(Time.SECONDS))
                waited += self.POLL_INTERVAL
              else:
                break
          
              if not self.is_alive:
                if self._popen_rc != 0:
                  raise TaskError('Task failed: %s' % self._popen_reason())
                else:
      >           log.info('Task runner exited: %s' % self._popen_reason())
      E           AttributeError: 'ThermosTaskRunner' object has no attribute '_popen_reason'
      
      /tmp/tmpxQyM2Q/apache/aurora/executor/thermos_task_runner.py:304: AttributeError
      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
      ====================================================================== 1 failed in 2.42 seconds ======================================================================
      src.test.python.apache.aurora.executor.thermos_task_runner                      .....   FAILURE
      

        Attachments

          Activity

            People

            • Assignee:
              wfarner Bill Farner
              Reporter:
              wfarner Bill Farner
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: