[AURORA-1724] Thermos runner fails if log rotation is enabled

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Assignee: John Sirois
• Reporter: Stephan Erb
• Created: 28/Jun/16 08:13
• Updated: 19/Sep/16 18:49
• Resolved: 01/Jul/16 19:53

Description

Thermos is failing to start when log rotation is enabled as described in our documentation

Steps to reproduce:

• start the latest developer vagrant box (or even then released 0.14)
• update to scheduler wide thermos_executor_flags to include --runner-logger-mode=rotate --runner-rotate-log-size-mb=100 --runner-rotate-log-backups=10 as documented (see above)
• launch a task
• observe that the task has failed with LOST : Runner died while task was active.

The thermos stderr then contains this traceback:

ERROR] Caught exception in self.control(): Logger mode rotate is invalid.
ERROR]   Traceback (most recent call last):
  File "apache/thermos/core/runner.py", line 572, in control
    yield
  File "apache/thermos/core/runner.py", line 916, in run
    self._run()
  File "apache/thermos/core/runner.py", line 923, in _run
    iteration_wait = runner.run()
  File "apache/thermos/core/runner.py", line 291, in run
    launched = self.runner._run_plan(self.runner._regular_plan)
  File "apache/thermos/core/runner.py", line 843, in _run_plan
    self._set_process_status(process_name, ProcessState.WAITING)
  File "apache/thermos/core/runner.py", line 687, in _set_process_status
    self._dispatcher.dispatch(self._state, runner_ckpt, self._recovery)
  File "apache/thermos/common/ckpt.py", line 384, in dispatch
    self._run_process_dispatch(process_update.state, process_update)
  File "apache/thermos/common/ckpt.py", line 215, in _run_process_dispatch
    getattr(handler, handler_function)(process_update)
  File "apache/thermos/core/runner.py", line 106, in on_waiting
    process_update.process, process_update.seq + 1))
  File "apache/thermos/core/runner.py", line 723, in _task_process_from_process_name
    preserve_env=self._preserve_env)
  File "apache/thermos/core/process.py", line 348, in __init__
    ProcessBase.__init__(self, *args, **kw)
  File "apache/thermos/core/process.py", line 142, in __init__
    raise ValueError("Logger mode %s is invalid." % self._logger_mode)
ValueError: Logger mode rotate is invalid.

What's kind of obscure here: "rotate" is one of the supported modes.

Issue Links

• is related to: AURORA-1730 Improve pystachio Logger schema