[SPARK-40499] Spark 3.2.1 percentlie_approx query much slower than Spark 2.4.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.2.1
• Component/s: Shuffle

Environment

hadoop: 3.0.0 

spark:  2.4.0 / 3.2.1

shuffle：spark 2.4.0

• Assignee: Unassigned
• Reporter: xuanzhiang
• Created: 20/Sep/22 08:57
• Updated: 12/Jan/24 23:21
• Resolved: 12/Oct/22 17:49

Description

spark.sql(
      s"""
         |SELECT
         | Info ,
         | PERCENTILE_APPROX(cost,0.5) cost_p50,
         | PERCENTILE_APPROX(cost,0.9) cost_p90,
         | PERCENTILE_APPROX(cost,0.95) cost_p95,
         | PERCENTILE_APPROX(cost,0.99) cost_p99,
         | PERCENTILE_APPROX(cost,0.999) cost_p999
         |FROM
         | textData
         |""".stripMargin)

• When we used spark 2.4.0, aggregation adopted objHashAggregator, stage 2 pull shuffle data very quick . but , when we use spark 3.2.1 and use old shuffle , 140M shuffle data cost 3 hours. 

• If we upgrade the Shuffle, will we get performance regression？

•  

Issue Links

• is related to: SPARK-46706 percentile_approx regression since Spark 2.4