[HDFS-12534] Provide logical BlockLocations for EC files for better split calculation

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-beta1
• Component/s: erasure-coding
• Labels: hdfs-ec-3.0-nice-to-have
• Target Version/s: 3.5.0
• Assignee: Unassigned
• Reporter: Andrew Wang
• Created: 22/Sep/17 23:54
• Updated: 04/Jan/24 07:59

Description

I talked to vanzin and alex.behm some more about split calculation with EC. It turns out HDFS-12222 was resolved prematurely. Applications depend on HDFS BlockLocation to understand where the split points are. The current scheme of returning one BlockLocation per block group loses this information.

We should change this to provide logical blocks. Divide the file length by the block size and provide suitable BlockLocations to match, with virtual offsets and lengths too.

I'm not marking this as incompatible, since changing it this way would in fact make it more compatible from the perspective of applications that are scheduling against replicated files. Thus, it'd be good for beta1 if possible, but okay for later too.