[BEAM-3908] Leaderboard / gamestats leaking Dataflow Jobs

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Affects Version/s: Not applicable
• Fix Version/s: 2.5.0
• Component/s: runner-dataflow, testing
• Assignee: Yifan Zou
• Reporter: Luke Cwik
• Created: 22/Mar/18 00:03
• Updated: 24/Jul/20 20:11
• Resolved: 11/Apr/18 17:38

Description

I found that the leaderboard/gamestats Dataflow streaming jobs weren't being cleaned up by the test infrastructure which lead to quota issues because all the VMs/disks/memory being consumed causing other Jenkins jobs to fail.

I manually stopped all the jobs that had been running for more then 12 hrs. There were about 20 jobs like this.

Example links:
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_22_19_25-7861256924404398606?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_49_54-7185486205606862436?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_23_14_26-6599347078760080693?project=apache-beam-testing&organizationId=433637338589
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_32_33-7276493109541122240?project=apache-beam-testing&organizationId=433637338589