[SAMZA-680] Invert the JobCoordinator and AM logic

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.1
• Assignee: Jagadish
• Reporter: Naveen Somasundaram
• Created: 15/May/15 23:14
• Updated: 02/Jun/16 21:55
• Resolved: 02/Jun/16 21:55

Description

Currently, the YARN AM pretty much dictates how the JobCoordinator works. This creates lot of inflexibility on how we can control failures or even integrate with new system (Mesos).
For e.g., https://issues.apache.org/jira/browse/SAMZA-465?focusedCommentId=14522043&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14522043

It would nice to invert the logic to JobCoordinator, and JobCoordinator has a global view of container failures, config changes etc. This simplifies lot of implementation specifics (for e.g., dynamic scaling becomes easier).

A another nice to have, would be make this logic pluggable.

e.g.,
job.clustermanager = org.apache.samza.YarnClusterManager
job.clustermanager = org.apache.samza.MesosClusterManager
 

Issue Links

• is related to: SAMZA-881 Re-think the Samza Job Coordinator