Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-781

ParquetOutputFormat should support custom OutputCommitter

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 1.14.0
    • None
    • parquet-mr
    • None
    • Patch

    Description

      ParquetOutputFormat should support custom OutputCommitter.

      There is a need to bypass current Hadoop functionality of writing output data under _temporary folder. Especially with AWS S3, there can be huge overhead of moving the files from _temporary folder to output folder.

      Attachments

        Issue Links

          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. PARQUET-2486 Improve Parquet IO Performance within cloud datalakes

          • Major - Major loss of function.
          • In Progress
          links to

          Web Link GitHub Pull Request #1361

          Activity

            People

              stevel@apache.org Steve Loughran
              mikkokupsu Mikko Kupsu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: