XMLWordPrintableJSON

Details

    • Sub-task
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.2.0
    • None
    • fs/s3
    • None

    Description

      s3a magic committer isRecoverySupported() is false, so will restart all task after application master restart for am jvm crashed, leaving pendingset in magic path not to clear. pendingset name format is jobAttemptPath + taskAttemptID.getTaskID() + ".pendingset", and jobAttemptPath is actually jobIdPath not JobAttemptIdPath in s3a magic committer. These pendingset files are overwritted by new task commit.

      But if in new am attempt, a speculative task overcomes origin task, so pendingset file in last attempt may be hold for job commit, the data for commit is wrong

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              adol yinan zhan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: