Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7062

Update mapreduce.job.tags description for making use for ATSv2 purpose.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.2.0, 3.0.3, 3.1.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When applications are submitted to YARN, tags are generated in the format 
      TIMELINE_FLOW_NAME_TAG:{flow_name},TIMELINE_FLOW_VERSION_TAG:{flow_version},TIMELINE_FLOW_RUN_ID_TAG:{flow_run_id}

      However, MR applications don't follow this format and the tags submitted via the property mapreduce.job.tags are of the format,

      {flow_name}

      ,{flow_version},{flow_run_id}

      Due to this, YARN falls back to default values for flow name, flow version and flow run id which in turn are used in ATSv2.

      There are 2 approaches that could be taken to make MR tags compatible with ATSv2,

      Fix in the MR code
      -------------------------
      Prefix any tags specified with the ones needed by the YARN Timeline Service v2. But MR is legacy code and hence these changes could affect how users are using these tags.

      Add a note in mapred-default.xml
      --------------------------------------------
      Add notes in the property name, mapreduce.job.tags mentioning that for purposes of ATSv2, prefixes need to be added to the tag names.

        Attachments

          Activity

            People

            • Assignee:
              charanh Charan Hebri
              Reporter:
              charanh Charan Hebri
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: