[YARN-8283] [Umbrella] MaWo - A Master Worker framework on top of YARN Services

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Yesha Vora
• Reporter: Yesha Vora
• Created: 11/May/18 22:39
• Updated: 08/Jan/20 23:25

Description

There is a need for an application / framework to handle Master-Worker scenarios. There are existing frameworks on YARN which can be used to run a job in distributed manner such as Mapreduce, Tez, Spark etc. But master-worker use-cases usually are force-fed into one of these existing frameworks which have been designed primarily around data-parallelism instead of generic Master Worker type of computations.

In this JIRA, we’d like to contribute MaWo - a YARN Service based framework that achieves this goal. The overall goal is to create an app that can take an input job specification with tasks, their durations and have a Master dish the tasks off to a predetermined set of workers. The components will be responsible for making sure that the tasks and the overall job finish in specific time durations.

We have been using a version of the MaWo framework for running unit tests of Hadoop in a parallel manner on an existing Hadoop YARN cluster. What typically takes 10 hours to run all of Hadoop project’s unit-tests can finish under 20 minutes on a MaWo app of about 50 containers!

YARN-3307 was an original attempt at this but through a first-class YARN app. In this JIRA, we instead use YARN Service for orchestration so that our code can focus on the core Master Worker paradigm.

Issue Links

• relates to: YARN-3307 Master-Worker Application on YARN

Sub-Tasks

• Build Worker utility for MaWo Application: Patch Available
• Build Master Job Module for MaWo Application: Patch Available
• Build Master module for MaWo app: Patch Available
• Fix MaWo_Config to read WORKER_WORK_SPACE and MASTER_TASKS_STATUS_LOG_PATH from env: Patch Available
• Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop Cluster: Patch Available