Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6608 Work Preserving AM Restart for MapReduce
  3. MAPREDUCE-6726

YARN Registry based AM discovery with retry and in-flight task persistent via JHS

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: applicationmaster
    • Labels:
      None

      Description

      Several tasks will be achieved in this JIRA based on the demo patch in MAPREDUCE-6608:
      1. AM discovery base on YARN register service. Could be replaced by YARN-4758 later due to scale up issue.
      2. Retry logic for TaskUmbilicalProtocol RPC connection
      3. In-flight task recover after AM restart via JHS
      4. Configuration to control the behavior compatible with previous when not enable this feature (by default).
      All security related issues and other concerns discussed in MAPREDUCE-6608 will be addressed in follow up JIRAs.

        Attachments

        1. MAPREDUCE-6726-MAPREDUCE-6608.001.patch
          131 kB
          Junping Du
        2. MAPREDUCE-6726-MAPREDUCE-6608.001.patch
          131 kB
          Srikanth Sampath
        3. MAPREDUCE-6726-MAPREDUCE-6608.002.patch
          121 kB
          Srikanth Sampath
        4. MAPREDUCE-6726-MAPREDUCE-6608.003.patch
          125 kB
          Srikanth Sampath
        5. WorkPreservingMRAppMaster.pdf
          112 kB
          Srikanth Sampath

          Activity

            People

            • Assignee:
              dmmkr D M Murali Krishna Reddy
              Reporter:
              junping_du Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated: