[SAMZA-2439] Remove LocalityManager and container location information from JobModel

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6
• Assignee: Bharath Kumarasubramanian
• Reporter: Prateek Maheshwari
• Created: 17/Jan/20 19:26
• Updated: 02/Feb/21 23:06
• Resolved: 02/Feb/21 23:06

Description

Goal: Make JobModel immutable by removing locality information (container to host mapping) from the JobModel.

Motivation:
1. Enables us to move JobModel to public APIs and expose it in JobContext.
2. Enables us to cache and serve serialized JobModel from the AM servlet to reduce AM overhead (memory, open connections, num threads) during container startup, esp. for jobs with a large number of containers (See: https://github.com/apache/samza/pull/1241)
3. Removes tech debt: models should be immutable, and should not update themselves.
4. Removes tech debt: makes current container location a first class concept for container scheduling / placement , and for tools like dashboard, samza-rest, auto-scaling, diagnostics etc.

Proposal:
Move locality information (and LocalityManager) out of the JobModel. New JobModel should only contain Config and ContainerModel.
Add a new AM HTTP endpoint for getting the container location information for external tools.
Update Samza Dashboard and any other tools that rely on locality information to get it from the new endpoint instead of the JobModel.

Issue Links

• links to: GitHub Pull Request #1421