[HADOOP-14943] Add common getFileBlockLocations() emulation for object stores, including S3A

• Type: Sub-task
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.8.1
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 11/Oct/17 14:27
• Updated: 04/Oct/22 16:23

Description

It looks suspiciously like S3A isn't providing the partitioning data needed in listLocatedStatus and getFileBlockLocations() needed to break up a file by the blocksize. This will stop tools using the MRv1 APIS doing the partitioning properly if the input format isn't doing it own split logic.

FileInputFormat in MRv2 is a bit more configurable about input split calculation & will split up large files. but otherwise, the partitioning is being done more by the default values of the executing engine, rather than any config data from the filesystem about what its "block size" is,

NativeAzureFS does a better job; maybe that could be factored out to hadoop-common and reused?

Issue Links

• breaks: SPARK-22240 S3 CSV number of partitions incorrectly computed
• contains: HADOOP-15044 Wasb getFileBlockLocations() returns too many locations.
• is depended upon by: HADOOP-15132 Über-jira: WASB client phase III: roll-up for Hadoop 3.2
• is related to: HADOOP-12878 Impersonate hosts in s3a for better data locality handling
• is related to: HADOOP-15000 s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet
• is related to: HADOOP-15320 Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake
• relates to: HDFS-12831 HDFS throws FileNotFoundException on getFileBlockLocations(path-to-directory)