Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6545

Test committer.commitJob() behavior during committing when MR AM get failed.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      In MAPREDUCE-5485, we are adding additional API (isCommitJobRepeatable) to allow job commit can tolerate AM failure in some cases (like FileOutputCommitter in v2 algorithm). Although we have unit test to cover most of flows, we may want a completed end to end test to verify the whole work flow.
      The scenario include:
      1. For FileOutputCommitter (or some sub class), emulate a MR AM failure or restart during commitJob() in progress
      2. Check different behavior for v1 and v2 (support isCommitJobRepeatable() or not)

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5485 Allow repeating job commit by extending OutputCommitter API

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              junping_du Junping Du
              junping_du Junping Du
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: