[SAMZA-719] Auto-scaling containers in Samza

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: design
• Assignee: Shadi A. Noghabi
• Reporter: Shadi A. Noghabi
• Created: 19/Jun/15 18:46
• Updated: 10/Oct/17 21:45

Description

Currently, a fixed number of containers is assigned to a job as an input configuration parameter. However, with this design jobs can fail due to lack of enough resources (such as memory), or they can become a bottleneck in a workflow containing many jobs. While auto-scaling is much broader term, the goal of this project will be to enable a Samza job to automatically scale its containers such that there is improved job performance.

Issue Links

• contains: SAMZA-755 Evaluate Auto-scaling In Memory Mode

Sub-Tasks

• Create a tool to read configuration changes after the the bootstrap of the job, and react to them accordingly: Resolved