[SPARK-48265] Infer window group limit batch should do constant folding

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.1
• Fix Version/s: 4.0.0, 3.5.2
• Component/s: SQL
• Labels: pull-request-available
• Assignee: angerszhu
• Reporter: angerszhu
• Created: 14/May/24 02:22
• Updated: 14/May/24 05:54
• Resolved: 14/May/24 05:45

Description

24/05/13 17:39:25 ERROR [main] PlanChangeLogger:
=== Result of Batch LocalRelation ===
 GlobalLimit 21                                                                                                                             GlobalLimit 21
 +- LocalLimit 21                                                                                                                           +- LocalLimit 21
!   +- Union false, false                                                                                                                      +- LocalLimit 21
!      :- LocalLimit 21                                                                                                                           +- Project [item_id#647L]
!      :  +- Project [item_id#647L]                                                                                                                  +- Filter (((isnotnull(tz_type#734) AND (tz_type#734 = local)) AND (grass_region#735 = BR)) AND isnotnull(grass_region#735))
!      :     +- Filter (((isnotnull(tz_type#734) AND (tz_type#734 = local)) AND (grass_region#735 = BR)) AND isnotnull(grass_region#735))               +- Relation db.table[,... 91 more fields] parquet
!      :        +- Relation db.table[,... 91 more fields] parquet
!      +- LocalLimit 21
!         +- Project [item_id#738L]
!            +- LocalRelation <empty>, [, ... 91 more fields]
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch Check Cartesian Products has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch RewriteSubquery has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch NormalizeFloatingNumbers has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch ReplaceUpdateFieldsExpression has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch Optimize Metadata Only Query has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch PartitionPruning has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch InjectRuntimeFilter has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch Pushdown Filters from PartitionPruning has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch Cleanup filters that cannot be pushed down has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger: Batch Extract Python UDFs has no effect.
24/05/13 17:39:25 ERROR [main] PlanChangeLogger:
=== Applying Rule org.apache.spark.sql.catalyst.optimizer.EliminateLimits ===
 GlobalLimit 21                                                                                                                          GlobalLimit 21
!+- LocalLimit 21                                                                                                                        +- LocalLimit least(, ... 2 more fields)
!   +- LocalLimit 21                                                                                                                        +- Project [item_id#647L]
!      +- Project [item_id#647L]                                                                                                               +- Filter (((isnotnull(tz_type#734) AND (tz_type#734 = local)) AND (grass_region#735 = BR)) AND isnotnull(grass_region#735))
!         +- Filter (((isnotnull(tz_type#734) AND (tz_type#734 = local)) AND (grass_region#735 = BR)) AND isnotnull(grass_region#735))            +- Relation db.table[,... 91 more fields] parquet
!            +- Relation db.table[,... 91 more fields] parquet
 

Issue Links

• links to: GitHub Pull Request #46568