[SPARK-27232] Ignore file locality in InMemoryFileIndex if spark.locality.wait is set to

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: EdisonWang
• Created: 21/Mar/19 15:59
• Updated: 12/Dec/22 18:10

Description

`InMemoryFileIndex` needs to request file block location information in order to do locality schedule in `TaskSetManager`.

Usually this is a time-cost task. For example, In our production env, there are 24 partitions, with totally 149925 files and 83TB in size. It costs about 10 minutes to request file block locations before submit a spark job. Even though I set `spark.sql.sources.parallelPartitionDiscovery.threshold` to 24 to make it parallelized, it also needs 2 minutes.

Anyway, this is a waste if we don't care about the locality of files(for example, storage and computation are separate).

So there should be a conf to control whether we need to send `getFileBlockLocations` request to HDFS NN. If user set `spark.locality.wait` to 0, file block location information is meaningless.

Here in this PR, if `spark.locality.wait` is set to 0, it will not request file location information anymore, which will save several seconds to minutes.

Issue Links

• links to: GitHub Pull Request #24175