[SPARK-31698] NPE on big dataset plans

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.4
• Component/s: Spark Core

Environment

AWS EMR: 30 machines, 7TB RAM total.

• Assignee: Unassigned
• Reporter: Viacheslav Tradunsky
• Created: 13/May/20 07:55
• Updated: 13/May/20 08:03
• Resolved: 13/May/20 08:03

Description

We have big dataset containing 275 SQL operations more than 275 joins.

On the terminal operation to write data, it fails with NullPointerException.

 

I understand that such big number of operations might not be what spark is designed for, but NullPointerException is not an ideal way to fail in this case.

 

For more details, please see the stacktrace.

Issue Links

• is fixed by: SPARK-29046 Possible NPE on SQLConf.get when SparkContext is stopping in another thread