[HADOOP-12041] Implement another Reed-Solomon coder in pure Java

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Kai Zheng
• Reporter: Kai Zheng
• Created: 29/May/15 02:50
• Updated: 12/May/16 18:24
• Resolved: 03/Feb/16 23:06

Description

Currently existing Java RS coders based on GaloisField implementation have some drawbacks or limitations:

• The decoder computes not really erased units unnecessarily (HADOOP-11871);
• The decoder requires parity units + data units order for the inputs in the decode API (HADOOP-12040);
• Need to support or align with native erasure coders regarding concrete coding algorithms and matrix, so Java coders and native coders can be easily swapped in/out and transparent to HDFS (HADOOP-12010);
• It's unnecessarily flexible but incurs some overhead, as HDFS erasure coding is totally a byte based data system, we don't need to consider other symbol size instead of 256.

This desires to implement another RS coder in pure Java, in addition to the existing GaliosField from HDFS-RAID. The new Java RS coder will be favored and used by default to resolve the related issues. The old HDFS-RAID originated coder will still be there for comparing, and converting old data from HDFS-RAID systems.

Issue Links

• depends upon: HADOOP-12040 Adjust inputs order for the decode API in raw erasure coder
• depends upon: HADOOP-12047 Indicate preference not to affect input buffers during coding in erasure coder
• depends upon: HADOOP-12327 Initialize output buffers with ZERO bytes in erasure coder
• is related to: HADOOP-12808 Rename the RS coder from HDFS-RAID as legacy