Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-17046

[Python] pyarrow.parquet.write_to_dataset fails to pass kwargs to write_table function

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 8.0.0
    • 9.0.0
    • Python

    Description

      According to PyArrow 8.0.0 documentation kwargs is "Additional kwargs for write_table function." However when I try to pass additional arguments, for example flavor, to the underlying write_table I get the following error

      TypeError: unexpected parquet write option: flavor

      This used to work in PyArrow versions as late as 7.0.0 but started to break in 8.0.0.

      Minimal example to reproduce the error

      import pyarrow as pa
      import pandas as pd
      import pyarrow.parquet as pq
      
      df = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})
      tb = pa.Table.from_pandas(df)
      pq.write_to_dataset(tb, "test.parquet", flavor="spark") 

      Attachments

        Issue Links

          Activity

            People

              mirkhosro Amir Khosroshahi
              mirkhosro Amir Khosroshahi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5.5h
                  5.5h

                  Slack

                    Issue deployment