[SAMZA-617] YARN host affinity in Samza

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Navina Ramesh
• Reporter: Navina Ramesh
• Created: 25/Mar/15 17:58
• Updated: 06/May/15 22:52

Description

Today in Samza we do not guarantee that a container gets deployed on the same machine upon a job upgrade/restart. Hence, the co-located data needs to restored every time a container restarts. Restoring data each time can be expensive, especially for applications that have a large data set.

If we can enable restarting containers on the same machine, we can re-use available local state.

Issue Links

• depends upon: SAMZA-618 Add container-to-host mapping messages to config stream
• depends upon: SAMZA-619 Modify Samza AppMaster to enable host-affinity
• depends upon: SAMZA-656 Script to periodically clean-up store directories that are persisted outside YARN
• depends upon: SAMZA-668 Document YARN host affinity in the website
• duplicates: SAMZA-335 Locality for Samza containers
• relates to: SAMZA-557 Reuse local state in SamzaContainer on clean shutdown
• relates to: SAMZA-598 Provide a tool to debug state stores
• relates to: SAMZA-670 Allow easier access to JMX port

Sub-Tasks

• Add metrics to gauge effectiveness of host-affinity solution: Resolved