[ARROW-8197] [Rust] DataFusion "create_physical_plan" returns incorrect schema?

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.15.1
• Fix Version/s: 0.17.0
• Component/s: Rust, Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24396
• Assignee: Andy Grove
• Reporter: Paddy Horan
• Created: 24/Mar/20 16:10
• Updated: 11/Jan/23 07:58
• Resolved: 25/Mar/20 13:42

Description

I am using DataFusion in a situation where I know there will only be a single file.  DataFusion currently collects all batches into a vector.

As I am writing the data back out I want to work with an iterator instead of a vector.

I have something as follows:

let plan = ctx.create_logical_plan(&sql).unwrap();
let plan = ctx.optimize(&plan).unwrap();
dbg!(plan.schema());  // Returns field names
let plan = ctx.create_physical_plan(&plan, batch_size).unwrap();
dbg!(plan.schema()); // Returns c0, c1, etc

Maybe this is expected after turning the plan into a physical plan?

I can change the schema of the returned batches, would this be the recommended way to address this or is there something in DataFusion I should leverage to do this?

Issue Links

• links to: GitHub Pull Request #6703