Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-5349

[Python/C++] Provide a way to specify the file path in parquet ColumnChunkMetaData

    XMLWordPrintableJSON

    Details

      Description

      After ARROW-5258 / https://github.com/apache/arrow/pull/4236 it is now possible to collect the file metadata while writing different files (then how to write those metadata was not yet addressed -> original issue ARROW-1983).

      However, currently, the file_path information in the ColumnChunkMetaData object is not set. This is, I think, expected / correct for the metadata as included within the single file; but for using the metadata in the combined dataset `_metadata`, it needs a file path set.

      So if you want to use this metadata for a partitioned dataset, there needs to be a way to specify this file path.
      Ideas I am thinking of currently: either, we could specify a file path to be used when writing, or expose the `set_file_path` method on the Python side so you can create an updated version of the metadata after collecting it.

      cc Pearu Peterson Martin Durant

        Attachments

        1. test_pyspark_dataset.zip
          5 kB
          Joris Van den Bossche

          Issue Links

            Activity

              People

              • Assignee:
                wesmckinn Wes McKinney
                Reporter:
                jorisvandenbossche Joris Van den Bossche
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 6h 10m
                  6h 10m