[SPARK-38166] Duplicates after task failure in dropDuplicates and repartition

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.2
• Component/s: SQL
• Labels: correctness

Environment

Cluster runs on K8s. AQE is enabled.

• Assignee: Unassigned
• Reporter: Willi Raschkowski
• Created: 09/Feb/22 12:36
• Updated: 09/Feb/22 17:35

Description

We're seeing duplicates after running the following

def compute_shipments(shipments):
    shipments = shipments.dropDuplicates(["ship_trck_num"])
    shipments = shipments.repartition(4)
    return shipments

and observing lost executors (OOMs) and task retries in the repartition stage.

We're seeing this reliably in one of our pipelines. But I haven't managed to reproduce outside of that pipeline. I'll attach driver logs - maybe you have ideas.