[HDFS-8425] [umbrella] Performance tuning, investigation and optimization for erasure coding

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: HDFS-7285
• Component/s: erasure-coding
• Tags: ec test
• Assignee: Unassigned
• Reporter: Rui Gao
• Created: 18/May/15 23:10
• Updated: 18/Jan/17 23:55

Description

This umbrella jira aims to track performance tuning, investigation and optimization for erasure coding.

Issue Links

• contains: HDFS-8426 Namenode shutdown for "ReplicationMonitor thread received Runtime exception"
• is related to: HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
• relates to: HDFS-8198 Erasure Coding: system test of TeraSort

Sub-Tasks

• Optimized handling of parity blocks in block management: Open
• Erasure coding: shrink size of indices in BlockInfoStriped: Open
• Handle slow writers for DFSStripedOutputStream: Open
• Erasure Coding: erasure coding chunk buffer allocation and management: In Progress
• Erasure Coding: BlockInfoStriped.removeStorage(..) move storage with same blockIndex to front slot: Patch Available
• DFSStripedOutputStream.writeChunk should not wait for writing parity: Patch Available
• Erasure Coding: Optimize block checksum recalculation logic on the fly by reconstructing multiple missed blocks at a time: Open
• Extend BlockSender to support multiple block data source: Open