[SPARK-19705] Preferred location supporting HDFS Cache for FileScanRDD

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.1.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: gagan taneja
• Shepherd: Sandy Ryza
• Created: 23/Feb/17 07:16
• Updated: 19/Jan/20 00:06
• Resolved: 21/May/19 04:16

Description

Although NewHadoopRDD and HadoopRdd considers HDFS cache while calculating preferredLocations, FileScanRDD do not take into account HDFS cache while calculating preferredLocations
The enhancement can be easily implemented for large files where FilePartition only contains single HDFS file
The enhancement will also result in significant performance improvement for cached hdfs partitions

Issue Links

• links to: [Github] Pull Request #17035 (tanejagagan)
• links to: GitHub Pull Request #17035