Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-8446

[Python][Dataset] Detect and use _metadata file in a list of file paths

    XMLWordPrintableJSON

    Details

      Description

      From https://github.com/dask/dask/pull/6047#discussion_r402391318

      When specifying a directory to ParquetDataset, we will detect if a _metadata file is present in the directory and use that to populate the metadata attribute (and not include this file in the list of "pieces", since it does not include any data).

      However, when passing a list of files to ParquetDataset, with one being "_metadata", the metadata attribute is not populated, and the "_metadata" path is included as one of the ParquetDatasetPiece objects instead (which leads to an ArrowIOError during the read of that piece).

      We could detect it in a list of paths as well.

      Note, I mentioned ParquetDataset, but if working on this, we should probably directly do it in the datasets API-based version.
      Also, I labeled this as Python and not C++ for now, as this might be something that can be handled on the Python side (once the C++ side knows how to process this kind of metadata -> ARROW-8062)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jorisvandenbossche Joris Van den Bossche
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: