Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-5915

[C++] [Python] Set up testing for backwards compatibility of the parquet reader

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: C++, Python
    • Labels:

      Description

      Given the recent parquet compat problems, we should have better testing for this.

      For easy testing of backwards compatibility, we could add some files (with different types) written with older versions, and ensure they are read correctly with the current version.

      Similarly as what Kartothek is doing: https://github.com/JDASoftwareGroup/kartothek/tree/master/reference-data/arrow-compat

      An easy way would be to do that in pyarrow and add them to /pyarrow/tests/data/parquet (we already have some files from 0.7 there).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jorisvandenbossche Joris Van den Bossche
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: