Executing a sql statement with a large number of partitions requires a high memory space for the driver even there are no requests to collect data back to the driver.
Here are steps to re-produce the issue.
1. Start spark shell with a spark.driver.maxResultSize setting
2. Execute the code
The error message is