[ARROW-9678] [Rust] [DataFusion] Improve projection push down to remove unused columns

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Rust, Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25734
• Assignee: Jorge Leitão
• Reporter: Jorge Leitão
• Created: 09/Aug/20 16:45
• Updated: 11/Jan/23 08:08
• Resolved: 22/Aug/20 16:41

Description

Currently, the projection push down only removes columns that are never referenced in the plan. However, sometimes a projection declares columns that themselves are never used.

This issue is about improving the projection push-down to remove any column that is not logically required by the plan.

Failing unit-test with the idea:

    #[test]
    fn table_unused_column() -> Result<()> {
        let table_scan = test_table_scan()?;
        assert_eq!(3, table_scan.schema().fields().len());
        assert_fields_eq(&table_scan, vec!["a", "b", "c"]);

        // we never use "b" in the first projection => remove it
        let plan = LogicalPlanBuilder::from(&table_scan)
            .project(vec![col("c"), col("a"), col("b")])?
            .filter(col("c").gt(&lit(1)))?
            .project(vec![col("c"), col("a")])?
            .build()?;

        assert_fields_eq(&plan, vec!["c", "a"]);

        let expected = "\
        Projection: #c, #a\
        \n  Selection: #c Gt Int32(1)\
        \n    Projection: #c, #a\
        \n      TableScan: test projection=Some([0, 2])";

        assert_optimized_plan_eq(&plan, expected);

        Ok(())
    }

This issue was firstly identified by andygrove here.

Issue Links

• links to: GitHub Pull Request #7919