Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9219

Unable to Invoke MapReduce from Yarn REST API

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: yarn
    • Labels:
      None
    • Environment:

      Vanilla Hadoop 2.9.2 and HDP 3+

      Description

      The following description shows how REST API can be used to invoke Yarn jobs:

      **https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_yarn-resource-management/content/ch_yarn_rest_apis.html

      But we are unable to do this to run MapReduce.
      We can't use YARN Resource Manager API(s) to execute (submit) map-reduce applications. As documented here YARN (RM) REST API(s) mechanism to use RM API(s) involve retrieving application ID and then submit the application. It works fine for the Spark Job but not really for MapReduce Jobs. 


      If we execute a map-reduce application using these API(s), on completion of the sub-process parent application fails with  "Application application_xx_00xx failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_xx_00xx_000001 exited with exitCode: 0". You can see in the image it will launch two tasks parent and sub-task. Even after successful completion of sub-task parent task reports a failure. 

      I think it is a bug. Yarn should be able to support REST Api access for MapReduce.

      Links where this problems are highlighted:
      http://riteshkumargupta.blogspot.com/2019/01/submit-mapreduce-jobs-to-yarn-rm-using.html
      https://stackoverflow.com/questions/33341185/run-a-mapreduce-job-via-rest-api

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Kabra Namit
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: