[SAMZA-1554] Host affinity in standalone.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 11/Jan/18 01:47
• Updated: 09/Apr/19 01:07
• Resolved: 09/Apr/19 01:07

Description

Samza framework enables its users to build stateful stream processing applications–that is, applications that remember information about past events in a local state(store), which will be then used to influence the processing of future events from the stream. Local state is a fundamental and enabling concept in stream processing which is required and essential to support a majority of common use cases such as stream-stream join, stream-table join, windowing etc.

Local store of a task instance is backed up by an log compacted kafka topic referred to as change-log. When a task instance commits, incremental local task store updates are flushed to the kafka topic. When a task instance runs on a host that doesn’t have latest local store, it’s restored by replaying messages from the change-log stream. For large stateful jobs, this restoration phase takes longer time, thus preventing the application from starting up and processing events from the input streams. Host affinity is a feature that maintains stickiness between a task and physical host and offers best-effort guarantees that a task instance will be assigned to run on the same physical it had ran before.

This tracks the work required to accomplish this feature.

Issue Links

• Dependency: SAMZA-1786 Introduce metadata store abstraction.
• Dependency: SAMZA-1788 Introduce LocationIdProvider abstraction
• links to: GitHub Pull Request #448
• mentioned in: Page Loading...