[HDFS-12222] Document and test BlockLocation for erasure-coded files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-beta1
• Labels: hdfs-ec-3.0-nice-to-have
• Target Version/s: 3.0.0-beta1
• Assignee: Huafeng Wang
• Reporter: Andrew Wang
• Created: 28/Jul/17 21:26
• Updated: 13/Sep/17 01:27
• Resolved: 13/Sep/17 00:35

Description

HDFS applications query block location information to compute splits. One example of this is FileInputFormat:

https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346

You see bits of code like this that calculate offsets as follows:

    long bytesInThisBlock = blkLocations[startIndex].getOffset() + 
                          blkLocations[startIndex].getLength() - offset;

EC confuses this since the block locations include parity block locations as well, which are not part of the logical file length. This messes up the offset calculation and thus topology/caching information too.

Applications can figure out what's a parity block by reading the EC policy and then parsing the schema, but it'd be a lot better if we exposed this more generically in BlockLocation instead.