Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-9778

[Rust] [DataFusion] Logical and physical schemas' nullability does not match in 8 out of 20 end-to-end tests

    XMLWordPrintableJSON

Details

    Description

      In `tests/sql.rs`, if we re-write the ```execute``` function to test the end schemas, as

      ```
      /// Execute query and return result set as tab delimited string
      fn execute(ctx: &mut ExecutionContext, sql: &str) -> Vec<String> {
      let plan = ctx.create_logical_plan(&sql).unwrap();
      let plan = ctx.optimize(&plan).unwrap();
      let physical_plan = ctx.create_physical_plan(&plan).unwrap();
      let results = ctx.collect(physical_plan.as_ref()).unwrap();
      if results.len() > 0

      { // results must match the logical schema assert_eq!(plan.schema().as_ref(), results[0].schema().as_ref()); }

      result_str(&results)
      }
      ```

      we end up with 8 tests failing, which indicates that our physical and logical plans are not aligned. In all cases, the issue is nullability: our logical plan assumes nullability = true, while our physical plan may change the nullability field.

      If we do not plan to track nullability on the logical level, we could consider replacing Schema by a type that does not track nullability.

      Attachments

        Issue Links

          Activity

            People

              andygrove Andy Grove
              jorgecarleitao Jorge Leitão
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h