Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3690 [Rust] Add Rust to the format integration testing
  3. ARROW-10261

[Rust] [BREAKING] Lists should take Field instead of DataType

    XMLWordPrintableJSON

Details

    Description

      There is currently no way of tracking nested field metadata on lists. For example, if a list's children are nullable, there's no way of telling just by looking at the Field.

      This causes problems with integration testing, and also affects Parquet roundtrips.

      I propose the breaking change of [Large|FixedSize]List taking a Field instead of Box<DataType>, as this will overcome this issue, and ensure that the Rust implementation passes integration tests.

      CC andygrove jorgecarleitao alamb  jhorstmann (carols10cents as this addresses some of the roundtrip failures).

      I'm leaning towards this landing in 3.0.0, as I'd love for us to have completed or made significant traction on the Arrow Parquet writer (and reader), and integration testing, by then.

      Attachments

        Issue Links

          Activity

            People

              nevi_me Neville Dipale
              nevi_me Neville Dipale
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m