Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-12904

[Rust] Unable to load Feather v2 files created by pyarrow and pandas.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Information Provided
    • 4.0.1
    • None
    • Rust
    • None
    • Ubuntu 20.04

    Description

      arrow-rs seems unable to load Feather v2 files created by pyarrow (and pandas), while it can read Feather v2 created by itself.

      More info at:

      https://github.com/apache/arrow-rs/issues/286

       

      Any idea what is missing in the Rust implementation (missing part of the spec?)?

       

      More details: in both files, I am getting the following:
      Reading Utf8
      field_node: FieldNode { length: 7, null_count: 0 }
      offset buffer: Buffer { offset: 200, length: 55 }
      offsets: [32, 0, 407708164, 545407072, 8388608, 67108864, 134217728, 201326592]
      values buffer: Buffer { offset: 256, length: 51 }
      
      
      
      offsets[0] != 0 indicates a problem: offsets are expected to start from zero on any array with offsets.
      offsets[i+1] < offsets[i+1] for some i, which indicates a problem: offsets are expected to be monotonically increasing
      
      I do not have a root cause yet, these are just observations.
      

      https://github.com/apache/arrow-rs/issues/286#issuecomment-839524898

       

       In the attachment the following files can be found.

      {{}}

      test_pandas.feather: Original Feather file
      test_arrow.feather: loading test_pandas.feather with pyarrow and saving with pyarrow: df_pa = pa.feather.read_feather('test_pandas.feather')
      test_polars.feather:  Loading test_pandas.feather with pyarrow and saving with polars (only this one can be read by arrow-rs)
      test_pandas_from_polars.feather: Loading test_polars.feather with polars and using the to_pandas option.
       
       
      

       

      [^test_feather_file.zip]

      Attachments

        Activity

          People

            jorisvandenbossche Joris Van den Bossche
            ghuls Gert Hulselmans
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: