[SAMZA-2741] Pipeline Drain: Support the ability to drain pipelines to allow incompatible intermediate schema changes

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.8
• Assignee: Ajo Thomas
• Reporter: Ajo Thomas
• Created: 10/May/22 16:58
• Updated: 16/Dec/22 17:29
• Resolved: 16/Dec/22 17:29

Description

Samza users running long running pipelines need a way to gracefully drain them in preparation for an upgrade. The upgrade could be due to changes to transforms or backward incompatible changes in data schema. The only option currently available to the user is to abruptly stop the pipeline and reset intermediate data which could lead to loss of buffered data. User logic is expected to constantly evolve and schema changes are often backward incompatible. It is therefore imperative to provide the ability to seamlessly stop and upgrade their pipelines without losing any buffered data.

Sub-Tasks

• [Pipeline Drain] Add DrainMonitor and DrainManager and integrate it with yarn JC and SamzaContainer: Resolved
• [Pipeline Drain] Runloop and High-Level API changes: Resolved
• [Pipeline Drain] Simplifying DrainUtils writeDrainNotification method and change tests: Resolved
• [Pipeline Drain] Add task callback timeout override for drain: Resolved
• [Pipeline Drain] Fix processing of Drain messages for both low-level and high level API: Resolved
• [PipelineDrain] ApplicationUtil.isHighLevelJob required by drain doesn't work for anonymous SamzaApplication: Resolved
• [PipelineDrain] Add drainMode to DrainNotification: Resolved
• [Pipeline Drain] Enable drain monitor by default: Resolved
• [Pipeline Drain] Enable drain monitor by default: Resolved