[HDFS-8031] Follow-on work for erasure coding phase I (striping layout)

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Zhe Zhang
• Created: 31/Mar/15 17:32
• Updated: 18/Jan/17 23:55

Issue Links

• depends upon: HDFS-7285 Erasure Coding Support inside HDFS
• incorporates: HDFS-7661 [umbrella] support hflush and hsync for erasure coded files
• is related to: HADOOP-11842 Common side change for follow-on work for erasure coding phase I
• is related to: HDFS-8030 HDFS Erasure Coding Phase II -- EC with contiguous layout
• relates to: HDFS-11347 [Umbrella] Converting files from replicated to striped EC
• relates to: HDFS-11348 [Umbrella] Append and truncate for striped EC files
• relates to: HDFS-7337 Configurable and pluggable erasure codec and policy
• relates to: HDFS-8425 [umbrella] Performance tuning, investigation and optimization for erasure coding
• relates to: HDFS-7955 Improve naming of classes, methods, and variables related to block replication and recovery
• relates to: HDFS-11349 [Umbrella] Additional testing for striped erasure coding

Sub-Tasks

• Block Readers and Writers used in both client side and datanode side: Resolved
• Erasure Coding: unifying common constructs like coding work, block reader and block writer across client and DataNode: Resolved
• WebHDFS: Support EC commands through webhdfs: Resolved
• Handle hflush and hsync in the best optimal way possible during online Erasure encoding: Resolved
• Erasure Coding: Add more EC zone management APIs (get/list EC zone(s)): Resolved
• Allow to configure the system default EC schema: Resolved
• ECManager should be able to manage multiple ECSchemas: Resolved
• Erasure Coding: Persist erasure coding policies in NameNode: Resolved
• Erasure coding: NameNode manages multiple erasure coding policies: Resolved
• [umbrella] Adding metrics for Erasure Coding: Resolved
• ECSchema supports for offline EditsVisitor over an OEV XML file: Resolved
• Erasure Coding: Expose refreshECSchemas command to reload predefined schemas: Resolved
• Erasure coding: revisit how to store EC schema and cellSize in NameNode: Resolved
• Add MODIFY and REMOVE ECSchema editlog operations: Resolved
• Erasure Coding: local and remote block reader for coding work in DataNode: Resolved
• Erasure Coding: Create FileStatus isErasureCoded() method: Resolved
• Erasure Coding: Update last cellsize calculation according to whether the erasure codec has chunk boundary: Resolved
• Erasure Coding: local and remote block writer for coding work in DataNode: Resolved
• [umbrella] Erasure Coding worker and support in DataNode: Resolved
• Erasure Coding: Correctly calculate last striped block length in DFSStripedInputStream if it's under construction.: Resolved
• Add computation time metrics to datanode for ECWorker: Resolved
• Add bytes count metrics to datanode for ECWorker: Resolved
• Erasure Coding: Allow concat striped files if they have the same ErasureCodingPolicy: Resolved
• Add tasks count metrics to datanode for ECWorker: Resolved
• Ec files can't be deleted into Trash because of that Trash isn't EC zone.: Resolved
• Relax permission checking for EC related operations: Resolved
• Merge HDFS-8227 into EC branch: Resolved
• Create EC zone should not need superuser privilege: Resolved
• Erasure Coding: optimize client writing by making the writing of data and parity concurrently: Resolved
• Add blocks count metrics to datanode for ECWorker: Resolved
• Erasure Coding: cache ErasureCodingZone: Resolved
• Remove hard-coded chunk size in favor of ECZone: Resolved
• Remove hard-coded values in favor of EC schema: Resolved
• Erasure Coding: update invalidateBlock(..) logic for striped block: Resolved
• Erasure Coding: use thread pool for StripedDataStreamer: Resolved
• Erasure Coding: revisit buffer used for encoding and decoding.: Resolved
• Erasure Coding: Add EC-related Metrics to NN (seperate striped blocks count from UnderReplicatedBlocks count): Resolved
• Erasure coding: use simple replication for internal blocks on decommissioning datanodes: Resolved
• Erasure Coding: Revisit the long and int datatypes usage in striping logic: Resolved
• Erasure Coding: block group ID displayed in WebUI is not consistent with fsck: Resolved
• Remove the use of hard-coded cell size value in balancer Dispatcher: Resolved
• Erasure coding: merge HDFS-8499 to EC branch and refactor BlockInfoStriped: Resolved
• Use ByteBuffer in striping positional read: Resolved
• Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping read (position and stateful): Resolved
• Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping write: Resolved
• Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping recovery on DataNode side: Resolved
• Refactor DFSInputStream#ReaderStrategy: Resolved
• Erasure Coding: the log of each streamer should show its index: Resolved
• Erasure coding: a comprehensive I/O throughput benchmark tool: Resolved
• Tolerate multiple failures in DFSStripedOutputStream: Resolved
• Erasure Coding: client fails to write large file when one datanode fails: Resolved
• Erasure Coding: add tests for taking snapshots on EC files: Resolved
• Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message: Resolved
• Erasure Coding: cover more test situations of datanode failure during client writing: Resolved
• Erasure Coding: Move DFSStripedIO stream related classes to hadoop-hdfs-client: Resolved
• Erasure Coding: Lease recovery for striped file: Resolved
• Add InterfaceAudience annotation to the erasure coding classes: Resolved
• Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers: Resolved
• Inconsistent default value of dfs.datanode.stripedread.buffer.size: Resolved
• Erasure coding: Add apache license header in TestFileStatusWithECPolicy.java: Resolved
• Erasure Coding: Skip encoding the data cells if all the parity data streamers are failed for the current block group: Resolved
• Wait previous ErasureCodingWork to finish before schedule another one: Resolved
• Erasure coding: client should update and commit block based on acknowledged size: Resolved
• Erasure coding: friendly log information for write operations with some failed streamers: Resolved
• Erasure coding: updateBlockForPipeline sometimes returns non-striped block for striped file: Resolved
• Erasure coding: some EC tests are missing timeout: Resolved
• Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent file should be handled properly: Resolved
• Parallel optimization of DFSStripedOutputStream#flushAllInternals( ): Resolved
• Erasure coding: an erasure codec throughput benchmark tool: Resolved
• Use byte array for internal block indices in a striped block: Resolved
• Erasure Coding: Wrong limit setting of target ByteBuffer: Resolved
• Move ErasureCodingPolicyManager to FSDirectory: Resolved
• getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory: Resolved
• ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block: Resolved
• Erasure Coding: allow to use multiple EC policies in striping related tests: Resolved
• Make existing DFSClient#getFileChecksum() work for striped blocks: Resolved
• Refactoring ErasureCodingWorker into smaller reusable constructs: Resolved
• Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]: Resolved
• Correctly update DataNode's scheduled block size when writing small EC file: Resolved
• Streamer threads may leak if failure happens when closing the striped outputstream: Resolved
• Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]: Resolved
• Correctly handle EC reconstruction work caused by not enough racks: Resolved
• Erasure Coding: Postpone the recovery work for a configurable time period: Resolved
• Erasure Coding: Improve few exception handling logic of ErasureCodingWorker: Resolved
• Erasure Coding: Improve exception handling in ErasureCodingWorker#ReconstructAndTransferBlock: Resolved
• Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data: Resolved
• BlockManager#countNodes should be able to detect duplicated internal blocks: Resolved
• BlockManager#chooseExcessReplicasStriped may weaken rack fault tolerance: Resolved
• Missing block exception should carry locatedBlocks information: Resolved
• shouldProcessOverReplicated should not count number of pending replicas: Resolved
• Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped block at the same time: Open
• Erasure Coding: Sort located striped blocks based on decommissioned states: Resolved
• Erasure Coding: support small cluster whose #DataNode < # (Blocks in a BlockGroup): Resolved
• StripedFileTestUtil#readAll flaky: Resolved
• Fix intermittent test failure of TestDataNodeErasureCodingMetrics: Resolved
• Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block: Resolved
• Allow only suitable storage policies to be set on striped files: Resolved
• BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy: Resolved
• Add EC policy and storage policy related usage summarization function to dfs du command: Resolved
• Erasure Coding: Document about the current allowed storage policies for EC Striped mode files: Resolved
• Erasure Coding: Add removeErasureCodingPolicy API: Resolved
• Correctly report missing EC blocks in FSCK: Resolved
• When there are unrecoverable ec block groups, Namenode Web UI shows "There are X missing blocks." but doesn't show the block names.: Resolved
• FBR processing may generate incorrect reportedBlock-blockGroup mapping: Resolved
• Switch from "raw" to "system" xattr namespace for erasure coding policy: Resolved
• BlockManager#isInNewRack should consider decommissioning nodes: Resolved
• Distcp should not copy replication factor if source file is erasure coded: Resolved
• fsck -list-corruptfileblocks does not report corrupt EC files: Resolved
• Report erasure coding policy of EC files in Fsck: Resolved
• OIV tool should make an EC file explicit: Resolved
• Ability to specify per-file EC policy at create time: Resolved
• Support an XOR policy XOR-2-1-64k in HDFS: Resolved
• Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats: Resolved
• Correct typos in native erasure coding dump code: Resolved
• Improve test coverage for ISA-L native coder: Open
• Add ability to unset and change directory EC policy: Resolved
• Provide replicated EC policy to replicate files: Resolved
• Document dfs.client.read.striped configuration in hdfs-default.xml: Resolved
• Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping: Resolved
• Report blockIds of internal blocks for EC files in Fsck: Resolved
• Support an erasure coding policy using RS 10 + 4: Resolved
• Enforce set of enabled EC policies on the NameNode: Resolved
• Call RawErasureEncoder and RawErasureDecoder release() methods: Resolved
• Erasure Coding: Support Parity Blocks placement onto same nodes hosting Data Blocks when DataNodes are insufficient: Resolved
• Support ErasureCoding section in OIV XML/ReverseXML: Resolved
• Add javadoc for storage policy and erasure coding policy: Resolved
• Inotify should support erasure coding policy op as replica meta change: Resolved
• Erasure Coding: Miss EC block after decommission and restart NN: Open
• Erasure Coding: metrics xmitsInProgress become to negative.: Resolved
• Erasure Coding: decommissioning datanodes cause replicate a large number of duplicate EC internal blocks: Resolved