Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5485

Allow repeating job commit by extending OutputCommitter API

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.1.0-beta
    • 2.8.0, 2.7.3, 3.0.0-alpha1
    • None
    • None
    • Reviewed
    • Hide
      Previously, the MR job will get failed if AM get restarted for some reason (like node failure, etc.) during its doing commit job no matter if AM attempts reach to the maximum attempts.
      In this improvement, we add a new API isCommitJobRepeatable() to OutputCommitter interface which to indicate if job's committer can do commitJob again if previous commit work is interrupted by NM/AM failures, etc. The instance of OutputCommitter, which support repeatable job commit (like FileOutputCommitter in algorithm 2), can allow AM to continue the commitJob() after AM restart as a new attempt.
      Show
      Previously, the MR job will get failed if AM get restarted for some reason (like node failure, etc.) during its doing commit job no matter if AM attempts reach to the maximum attempts. In this improvement, we add a new API isCommitJobRepeatable() to OutputCommitter interface which to indicate if job's committer can do commitJob again if previous commit work is interrupted by NM/AM failures, etc. The instance of OutputCommitter, which support repeatable job commit (like FileOutputCommitter in algorithm 2), can allow AM to continue the commitJob() after AM restart as a new attempt.

    Description

      There are chances MRAppMaster crush during job committing,or NodeManager restart cause the committing AM exit due to container expire.In these cases ,the job will fail.
      However,some jobs can redo commit so failing the job becomes unnecessary.
      Let clients tell AM to allow redo commit or not is a better choice.
      This idea comes from Jason Lowe's comments in MAPREDUCE-4819

      Attachments

        1. MAPREDUCE-5485-demo.patch
          14 kB
          Junping Du
        2. MAPREDUCE-5485-demo-2.patch
          18 kB
          Junping Du
        3. MAPREDUCE-5485-v1.patch
          29 kB
          Junping Du
        4. MAPREDUCE-5485-v2.patch
          33 kB
          Junping Du
        5. MAPREDUCE-5485-v3.1.patch
          24 kB
          Junping Du
        6. MAPREDUCE-5485-v3.patch
          24 kB
          Junping Du
        7. MAPREDUCE-5485-v4.1.patch
          32 kB
          Junping Du
        8. MAPREDUCE-5485-v4.patch
          32 kB
          Junping Du
        9. MAPREDUCE-5485-v5.patch
          33 kB
          Junping Du
        10. MAPREDUCE-5485-v5-branch-2.7.patch
          33 kB
          Junping Du

        Issue Links

          Activity

            People

              junping_du Junping Du
              nemon Nemon Lou
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: