Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15220 Über-jira: S3a phase V: Hadoop 3.2 features
  3. HADOOP-15107

Stabilize/tune S3A committers; review correctness & docs

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.1.0
    • 3.1.2
    • fs/s3
    • None

    Description

      I'm writing about the paper on the committers, one which, being a proper paper, requires me to show the committers work.

      1. define the requirements of a "Correct" committed job (this applies to the FileOutputCommitter too)
      2. show that the Staging committer meets these requirements (most of this is implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset lists from committed tasks to the final destination, where they are read and committed.
      3. Show the magic committer also works.

      Attachments

        1. HADOOP-15107-001.patch
          35 kB
          Steve Loughran
        2. HADOOP-15107-002.patch
          44 kB
          Steve Loughran
        3. HADOOP-15107-003.patch
          42 kB
          Steve Loughran
        4. HADOOP-15107-004.patch
          44 kB
          Steve Loughran

        Issue Links

        contains

        Sub-task - The sub-task of the issue HADOOP-15095 S3a committer factory to warn when default FileOutputFormat committer is created

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Sub-task - The sub-task of the issue HADOOP-15470 S3A staging committers to not log FNFEs on job abort listings

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        is related to

        Sub-task - The sub-task of the issue HADOOP-15782 Clarify committers.md around v2 failure handling

        • Major - Major loss of function.
        • Open

        Sub-task - The sub-task of the issue HADOOP-15890 Some S3A committer tests don't match ITest* pattern; don't run in maven

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stevel@apache.org Steve Loughran
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment