Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6608 Work Preserving AM Restart for MapReduce
  3. MAPREDUCE-6726

YARN Registry based AM discovery with retry and in-flight task persistent via JHS

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • 3.2.2
    • applicationmaster
    • None

    Description

      Several tasks will be achieved in this JIRA based on the demo patch in MAPREDUCE-6608:
      1. AM discovery base on YARN register service. Could be replaced by YARN-4758 later due to scale up issue.
      2. Retry logic for TaskUmbilicalProtocol RPC connection
      3. In-flight task recover after AM restart via JHS
      4. Configuration to control the behavior compatible with previous when not enable this feature (by default).
      All security related issues and other concerns discussed in MAPREDUCE-6608 will be addressed in follow up JIRAs.

      Attachments

        1. MAPREDUCE-6726.004.patch
          229 kB
          D M Murali Krishna Reddy
        2. MAPREDUCE-6726.005.trunk.patch
          229 kB
          D M Murali Krishna Reddy
        3. MAPREDUCE-6726-MAPREDUCE-6608.001.patch
          131 kB
          Junping Du
        4. MAPREDUCE-6726-MAPREDUCE-6608.001.patch
          131 kB
          Srikanth Sampath
        5. MAPREDUCE-6726-MAPREDUCE-6608.002.patch
          121 kB
          Srikanth Sampath
        6. MAPREDUCE-6726-MAPREDUCE-6608.003.patch
          125 kB
          Srikanth Sampath
        7. TestReport.pdf
          30 kB
          D M Murali Krishna Reddy
        8. WorkPreservingMRAppMaster.pdf
          112 kB
          Srikanth Sampath

        Activity

          People

            dmmkr D M Murali Krishna Reddy
            junping_du Junping Du
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated: