[SAMZA-850] Yarn Job Validation Tool

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.1
• Assignee: Xinyu Liu
• Reporter: Xinyu Liu
• Created: 05/Jan/16 01:11
• Updated: 30/Mar/16 00:51
• Resolved: 30/Mar/16 00:51

Description

We've seen a strong need for validating running samza jobs programmatically. Consider the following scenarios:
1) running samza as a service (as in SAMZA-849), which requires automated job validation when upgrading the framework for large number of jobs;
2) production monitoring, automated validation can run periodically and trigger alerts;
3) integration testing, which can use validation tool to check the job status.

So a tool is needed to provide the checkout of job health. Minimally it will check:

• job submission
• app running
• container count

It should also be able to extend and validate the job metrics through metrics system, such as JMX. The user will be able to validate against important metrics such as process_count or message_behind_high_watermark.

Sub-Tasks

• Capture the details of Yarn Job Validation Tool in samza website: Open