[SPARK-20255] FileIndex hierarchy inconsistency

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Adrian Ionescu
• Reporter: Adrian Ionescu
• Created: 07/Apr/17 17:30
• Updated: 15/May/18 10:49
• Resolved: 07/Apr/17 21:00

Description

Trying to get a grip on the FileIndex hierarchy, I was confused by the following inconsistency:

On the one hand, PartitioningAwareFileIndex defines leafFiles and leafDirToChildrenFiles as abstract, but on the other it fully implements listLeafFiles which does all the listing of files. However, the latter is only used by InMemoryFileIndex.

I'm hereby proposing to move this method (and all its dependencies) to the implementation class that actually uses it, and thus unclutter the PartitioningAwareFileIndex interface.

Issue Links

• is related to: SPARK-24280 Speed up indexing of files in object stores by using listFiles(path, recursive=true)
• links to: [Github] Pull Request #17570 (adrian-ionescu)