[HUDI-1529] Spark-SQL drvier runs out of memory when metadata table is enabled

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Component/s: performance, spark
• Labels: pull-request-available
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: Udit Mehrotra
• Reporter: Udit Mehrotra
• Created: 15/Jan/21 00:13
• Updated: 04/Jan/22 00:09
• Resolved: 19/Jan/21 04:25

Description

When testing a large dataset around 1.2TB data and around 20k files, we notice an issue where the spark driver would always run out of memory, when running queries with use of metadata table enabled. The OOM would happen on any query, even if it was touching a single partition, and was happening in the split generation phase before any tasks would start executing.

Upon analyzing the heap dump, it was analyzed that input format was generating millions of splits for every single file. Upon further analysis of the code path, it was found that the root cause was because metadata enabled code was ignoring the blockSize when returning FileStatus objects and setting it to 0. Spark by itself does not set any value for the property:

mapreduce.input.fileinputformat.split.minsize

As a result minSize ends up being 1, and with block size as 0 it cause input format to generate splits of size 1 bytes**** because of the logic here:

https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L417

This ends up in exponential file split objects being creating, causing driver to run out of memory.

Issue Links

• links to: GitHub Pull Request #2451