Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3690 [Rust] Add Rust to the format integration testing
  3. ARROW-10261

[Rust] [BREAKING] Lists should take Field instead of DataType

    XMLWordPrintableJSON

    Details

      Description

      There is currently no way of tracking nested field metadata on lists. For example, if a list's children are nullable, there's no way of telling just by looking at the Field.

      This causes problems with integration testing, and also affects Parquet roundtrips.

      I propose the breaking change of [Large|FixedSize]List taking a Field instead of Box<DataType>, as this will overcome this issue, and ensure that the Rust implementation passes integration tests.

      CC Andy Grove Jorge Leitão Andrew Lamb  Jörn Horstmann (Carol Nichols as this addresses some of the roundtrip failures).

      I'm leaning towards this landing in 3.0.0, as I'd love for us to have completed or made significant traction on the Arrow Parquet writer (and reader), and integration testing, by then.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                nevi_me Neville Dipale
                Reporter:
                nevi_me Neville Dipale
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m