Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-6697

[Rust] [DataFusion] Validate that all parquet partitions have the same schema

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Rust, Rust - DataFusion
    • Labels:
      None

      Description

      When reading a partitioned Parquet file in DataFusion, the schema is read from the first partition and it is assumed that all other partitions have the same schema.

      It would be better to actually validate that all of the partitions have the same schema since there is no support for schema merging yet.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              andygrove Andy Grove
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: