[SPARK-36424] Support eliminate limits in AQE Optimizer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: XiDuo You
• Reporter: XiDuo You
• Created: 05/Aug/21 09:11
• Updated: 27/Jan/22 20:03
• Resolved: 09/Aug/21 08:52

Description

In Ad-hoc scenario, we always add limit for the query if user have no special limit value, but not all limit is nesessary.

With the power of AQE, we can eliminate limits using running statistics.

Issue Links

• links to: [Github] Pull Request #33651 (ulysses-you)
• links to: [Github] Pull Request #35301 (wangyum)
• links to: [Github] Pull Request #35301 (wangyum)