Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16546

make sure staging committers collect DTs for the staging FS

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.2.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      This is not a problem I've seen in the wild, but I've now encountered a problem with hive doing something like this

      we need to (somehow) make sure that the staging committers collect DTs for the staging dir FS. If this is the default FS or the same as a source or dest FS, this is handled elsewhere, but otherwise we need to add the staging fs.

      I don;t see an easy way to do this, but we could add a new method to PathOutputCommitter to collect DTs; FileOutputFormat can invoke this alongside its ongoing collection of tokens for the output FS. Base impl would be a no-op, obviously.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: