[YARN-8913] Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop Cluster

• Type: Sub-task
• Status: Patch Available
• Priority: Blocker
• Resolution: Unresolved
• Assignee: Yesha Vora
• Reporter: Yesha Vora
• Created: 18/Oct/18 21:45
• Updated: 21/May/19 14:44

Description

MaWo application can be used to run Hadoop UT faster in a Hadoop cluster.
Develop helper scripts to orchestrate end-to-end workflow for running Hadoop UT using MaWo app.

Pre-requisite:

• A Hadoop Cluster with HDFS and YARN installed
• Enable Docker on YARN feature

 

Helper-scripts

• MaWo_Driver
  • create a docker image with latest hadoop source code
  • create payload to MaWo app (This is input to mawo app where Each MaWo Task = UT execution of each Hadoop Module)
  • Upload payload file to HDFS
  • Update MaWo-Launch.json to resolve RM_HOST / Docker Image etc dynamically
  • Launch MaWo app in Hadoop cluster