[SPARK-50386] Improve SparkFatalException Propagation when OutOfMemoryError occurs on BroadcastExchangeExec building small table to broadcast

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.4, 3.5.3
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Eren Avsarogullari
• Created: 5 days ago 22:03
• Updated: 5 days ago 22:24

Description

When BroadcastHashJoin builds small table to broadcast by BroadcastExchangeExec, if OutOfMemoryError has occurred on driver, BroadcastExchangeExec throws SparkFatalException which wraps SparkException. However, SparkException' s cause property comes as null by missing actual cause which is `java.lang.OutOfMemoryError: Java heap space` in following example. Actual cause info is also useful to propagate Throwable.getCause / getCause.* properties to clients. Repro test case has been added.

Issue Links

• links to: GitHub Pull Request #48925