Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks
  3. MAPREDUCE-5956

MapReduce AM should not use maxAttempts to determine if this is the last retry

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 2.4.0
    • 2.6.0
    • applicationmaster, mrv2
    • None
    • Reviewed

    Description

      Found this while reviewing YARN-2074. The problem is that after YARN-2074, we don't count AM preemption towards AM failures on RM side, but MapReduce AM itself checks the attempt id against the max-attempt count to determine if this is the last attempt.

          public void computeIsLastAMRetry() {
            isLastAMRetry = appAttemptID.getAttemptId() >= maxAppAttempts;
          }
      

      This causes issues w.r.t deletion of staging directory etc..

      Attachments

        1. MR-5956.patch
          14 kB
          Wangda Tan
        2. MR-5956.patch
          20 kB
          Wangda Tan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            leftnoteasy Wangda Tan
            vinodkv Vinod Kumar Vavilapalli
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment