Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
There is a need for an application / framework to handle Master-Worker scenarios. There are existing frameworks on YARN which can be used to run a job in distributed manner such as Mapreduce, Tez, Spark etc. But master-worker use-cases usually are force-fed into one of these existing frameworks which have been designed primarily around data-parallelism instead of generic Master Worker type of computations.
In this JIRA, we’d like to contribute MaWo - a YARN Service based framework that achieves this goal. The overall goal is to create an app that can take an input job specification with tasks, their durations and have a Master dish the tasks off to a predetermined set of workers. The components will be responsible for making sure that the tasks and the overall job finish in specific time durations.
We have been using a version of the MaWo framework for running unit tests of Hadoop in a parallel manner on an existing Hadoop YARN cluster. What typically takes 10 hours to run all of Hadoop project’s unit-tests can finish under 20 minutes on a MaWo app of about 50 containers!
YARN-3307 was an original attempt at this but through a first-class YARN app. In this JIRA, we instead use YARN Service for orchestration so that our code can focus on the core Master Worker paradigm.
Attachments
Attachments
Issue Links
- relates to
-
-
- Open
-
1.
|
Build Common module for MaWo application |
|
Resolved | Yesha Vora |
2.
|
Build Worker utility for MaWo Application |
|
Patch Available | Yesha Vora |
3.
|
Build Master Job Module for MaWo Application |
|
Patch Available | Yesha Vora |
4.
|
Build Master module for MaWo app |
|
Patch Available | Yesha Vora |
5.
|
Fix MaWo_Config to read WORKER_WORK_SPACE and MASTER_TASKS_STATUS_LOG_PATH from env |
|
Patch Available | Yesha Vora |
6.
|
Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop Cluster |
|
Patch Available | Yesha Vora |
7.
|
ArtifactId of MaWo application is wrong |
|
Resolved | Wanqiang Ji |
8.
|
Sync Mockito version with other modules |
|
Resolved | Adam Antal |