[SPARK-28528] Fix Idempotence for Once batches in Catalyst optimizer

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yesheng Ma
• Created: 26/Jul/19 17:49
• Updated: 07/Aug/19 21:45

Description

As per https://github.com/apache/spark/pull/25249

Issue Links

• links to: GitHub Pull Request #25381

Sub-Tasks

• Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence: Resolved
• Fix Join Reorder optimizer rule to enforce idempotence: Resolved
• Improve Extract Python UDFs optimizer rule to enforce idempotence: Open
• Fix subquery optimizer rule to enforce idempotence: Resolved
• Idempotence checker for Idempotent batches in RuleExecutors: Resolved
• Once optimizer rule NormalizeFloatingNumbers is not idempotent: Resolved
• Improve partition pruning rule to enforce idempotence: Open