Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8283 [Umbrella] MaWo - A Master Worker framework on top of YARN Services
  3. YARN-8913

Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop Cluster

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      MaWo application can be used to run Hadoop UT faster in a Hadoop cluster.
      Develop helper scripts to orchestrate end-to-end workflow for running Hadoop UT using MaWo app.

      Pre-requisite:

      • A Hadoop Cluster with HDFS and YARN installed
      • Enable Docker on YARN feature

       

      Helper-scripts

      • MaWo_Driver
        • create a docker image with latest hadoop source code
        • create payload to MaWo app (This is input to mawo app where Each MaWo Task = UT execution of each Hadoop Module)
        • Upload payload file to HDFS
        • Update MaWo-Launch.json to resolve RM_HOST / Docker Image etc dynamically
        • Launch MaWo app in Hadoop cluster

        Attachments

        1. YARN-8913.001.patch
          45 kB
          Yesha Vora
        2. YARN-8913.002.patch
          51 kB
          Yesha Vora
        3. YARN-8913.003.patch
          51 kB
          Yesha Vora

          Activity

            People

            • Assignee:
              yeshavora Yesha Vora
              Reporter:
              yeshavora Yesha Vora
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: