[SPARK-49950] spark planing is too slow when LocalRelation has a lot of data

• Type: Improvement
• Status: Open
• Priority: Trivial
• Resolution: Unresolved
• Affects Version/s: 3.5.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: xie shuiahu
• Created: 14/Oct/24 04:28
• Updated: 14/Oct/24 09:54

Description

```scala

import spark.implicits._

val data = (0 until 10000000).toArray

 

 // spend more than 10s

val ds = spark.createDataset(data) 

// spend more than 5s

ds.selectExpr("value+2 as value").selectExpr("value+3 as value").selectExpr("value+4 as value").selectExpr("value+5 as value") 

```

 

This is caused by `LocalRelation`, because `mapExpressions` will go into the data and spend a lot of time to tranversal it. Any idea to fix this issue?