[AURORA-1155] GC executor broken / Insufficient automated testing

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Sprint: Twitter Aurora Q1'15 Sprint 3
• Assignee: Brian Wickman
• Reporter: Stephan Erb
• Created: 26/Feb/15 12:40
• Updated: 04/May/15 13:12
• Resolved: 26/Feb/15 22:48

Description

Last week, we deployed git revision e0e3f2e (https://github.com/apache/incubator-aurora/tree/e0e3f2e) onto our test cluster.

For this revision, all GC executor runs are failing:

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0224 15:00:23.165495 26402 fetcher.cpp:76] Fetching URI '/opt/thermos/bin/gc_executor.pex'
I0224 15:00:23.165679 26402 fetcher.cpp:179] Copying resource from '/opt/thermos/bin/gc_executor.pex' to '/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc'
Traceback (most recent call last):
  File "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/.bootstrap/_pex/pex.py", line 272, in execute
    self.execute_entry(entry_point, args)
  File "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/.bootstrap/_pex/pex.py", line 320, in execute_entry
    runner(entry_point)
  File "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/.bootstrap/_pex/pex.py", line 342, in execute_pkg_resources
    runner = entry.load(require=False)  # trust that the environment is sane
  File "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/.bootstrap/pkg_resources.py", line 2048, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/apache/aurora/executor/bin/gc_executor_main.py", line 27, in <module>
ImportError: No module named executor_detector

For me this is an indication that the test infrastructure is missing an integration or end to end tests which thoroughly tests the GC executor.