[ARROW-9809] [Rust] [DataFusion] logical schema = physical schema is not true

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Rust, Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25852
• Assignee: Jorge Leitão
• Reporter: Jorge Leitão
• Created: 20/Aug/20 06:20
• Updated: 11/Jan/23 08:09
• Resolved: 23/Aug/20 14:07

Description

In tests/sql.rs, we test that the physical and the optimized schema must match. However, this is not necessarily true for all our queries. An example:

#[test]
fn csv_query_sum_cast() {
    let mut ctx = ExecutionContext::new();
    register_aggregate_csv_by_sql(&mut ctx);
    // c8 = i32; c9 = i64
    let sql = "SELECT c8 + c9 FROM aggregate_test_100";
    // check that the physical and logical schemas are equal
    execute(&mut ctx, sql);
}

The physical expression (and schema) of this operation, after optimization, is CAST(c8 as Int64) Plus c9 (this test fails).

AFAIK, the invariant of the optimizer is that the output types and nullability are the same.

Also, note that the reason the optimized logical schema equals the logical schema is that our type coercer does not change the output names of the schema, even though it re-writes logical expressions. I.e. after the optimization, `.to_field()` of an expression may no longer match the field name nor type in the Plan's schema. IMO this is currently by (implicit?) design, as we do not want our logical schema's column names to change during optimizations, or all column references may point to non-existent columns. This is something that brought up on the mailing list about polymorphism.

Issue Links

• duplicates: ARROW-4957 [Rust] [DataFusion] Implement get_supertype correctly
• links to: GitHub Pull Request #8024