[SPARK-35421] Remove redundant ProjectExec from streaming queries with V2Relation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Structured Streaming
• Assignee: Kousuke Saruta
• Reporter: Kousuke Saruta
• Created: 17/May/21 08:13
• Updated: 19/May/21 06:19
• Resolved: 19/May/21 06:19

Description

Streaming queries with V2Relation can have redundant ProjectExec in it's physical plan.
You can easily reproduce with the following code.

import org.apache.spark.sql.streaming.Trigger

val query = spark.
  readStream.
  format("rate").
  option("rowsPerSecond", 1000).
  option("rampUpTime", "10s").
  load().
  selectExpr("timestamp", "100",  "value").  
  writeStream.
  format("console").
  trigger(Trigger.ProcessingTime("5 seconds")).
  // trigger(Trigger.Continuous("5 seconds")). // You can reproduce with continuous processing too.
  outputMode("append").
  start()

Issue Links

• links to: [Github] Pull Request #32570 (sarutak)