Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5485

Allow repeating job commit by extending OutputCommitter API

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Previously, the MR job will get failed if AM get restarted for some reason (like node failure, etc.) during its doing commit job no matter if AM attempts reach to the maximum attempts.
      In this improvement, we add a new API isCommitJobRepeatable() to OutputCommitter interface which to indicate if job's committer can do commitJob again if previous commit work is interrupted by NM/AM failures, etc. The instance of OutputCommitter, which support repeatable job commit (like FileOutputCommitter in algorithm 2), can allow AM to continue the commitJob() after AM restart as a new attempt.
      Show
      Previously, the MR job will get failed if AM get restarted for some reason (like node failure, etc.) during its doing commit job no matter if AM attempts reach to the maximum attempts. In this improvement, we add a new API isCommitJobRepeatable() to OutputCommitter interface which to indicate if job's committer can do commitJob again if previous commit work is interrupted by NM/AM failures, etc. The instance of OutputCommitter, which support repeatable job commit (like FileOutputCommitter in algorithm 2), can allow AM to continue the commitJob() after AM restart as a new attempt.

      Description

      There are chances MRAppMaster crush during job committing,or NodeManager restart cause the committing AM exit due to container expire.In these cases ,the job will fail.
      However,some jobs can redo commit so failing the job becomes unnecessary.
      Let clients tell AM to allow redo commit or not is a better choice.
      This idea comes from Jason Lowe's comments in MAPREDUCE-4819

        Attachments

        1. MAPREDUCE-5485-v5-branch-2.7.patch
          33 kB
          Junping Du
        2. MAPREDUCE-5485-v5.patch
          33 kB
          Junping Du
        3. MAPREDUCE-5485-v4.1.patch
          32 kB
          Junping Du
        4. MAPREDUCE-5485-v4.patch
          32 kB
          Junping Du
        5. MAPREDUCE-5485-v3.1.patch
          24 kB
          Junping Du
        6. MAPREDUCE-5485-v3.patch
          24 kB
          Junping Du
        7. MAPREDUCE-5485-v2.patch
          33 kB
          Junping Du
        8. MAPREDUCE-5485-v1.patch
          29 kB
          Junping Du
        9. MAPREDUCE-5485-demo-2.patch
          18 kB
          Junping Du
        10. MAPREDUCE-5485-demo.patch
          14 kB
          Junping Du

          Issue Links

            Activity

              People

              • Assignee:
                djp Junping Du
                Reporter:
                nemon Nemon Lou
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: