Bulk Operation

HDFS-8031 Erasure Coding: decommissioning datanodes cause replicate a large number of duplicate EC internal blocks

HDFS-8031 Erasure Coding: metrics xmitsInProgress become to negative.

HDFS-8031 Erasure Coding: Miss EC block after decommission and restart NN

HDFS-8031 Inotify should support erasure coding policy op as replica meta change

HDFS-8031 Add javadoc for storage policy and erasure coding policy

HDFS-8031 Erasure Coding: Support Parity Blocks placement onto same nodes hosting Data Blocks when DataNodes are insufficient

HDFS-8031 Call RawErasureEncoder and RawErasureDecoder release() methods

HDFS-8031 Support ErasureCoding section in OIV XML/ReverseXML

HDFS-8031 Enforce set of enabled EC policies on the NameNode

HDFS-8031 Support an erasure coding policy using RS 10 + 4

HDFS-8031 Report blockIds of internal blocks for EC files in Fsck

HDFS-8031 Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping

HDFS-8031 Document dfs.client.read.striped configuration in hdfs-default.xml

HDFS-8031 Provide replicated EC policy to replicate files

HDFS-8031 Add ability to unset and change directory EC policy

HDFS-8031 Improve test coverage for ISA-L native coder

HDFS-8031 Correct typos in native erasure coding dump code

HDFS-8031 Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats

HDFS-8031 Ability to specify per-file EC policy at create time

HDFS-8031 Support an XOR policy XOR-2-1-64k in HDFS

HDFS-8031 OIV tool should make an EC file explicit

HDFS-8031 Report erasure coding policy of EC files in Fsck

HDFS-8031 fsck -list-corruptfileblocks does not report corrupt EC files

HDFS-8031 Distcp should not copy replication factor if source file is erasure coded

HDFS-8031 BlockManager#isInNewRack should consider decommissioning nodes

HDFS-8031 Switch from "raw" to "system" xattr namespace for erasure coding policy

HDFS-8031 FBR processing may generate incorrect reportedBlock-blockGroup mapping

HDFS-8031 When there are unrecoverable ec block groups, Namenode Web UI shows "There are X missing blocks." but doesn't show the block names.

HDFS-8031 Correctly report missing EC blocks in FSCK

HDFS-8031 Erasure Coding: Add removeErasureCodingPolicy API

HDFS-8031 Erasure Coding: Document about the current allowed storage policies for EC Striped mode files

HDFS-8031 Add EC policy and storage policy related usage summarization function to dfs du command

HDFS-8031 BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy

HDFS-8031 Allow only suitable storage policies to be set on striped files

HDFS-8031 Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block

HDFS-8031 Fix intermittent test failure of TestDataNodeErasureCodingMetrics

HDFS-8031 StripedFileTestUtil#readAll flaky

HDFS-8031 Erasure Coding: support small cluster whose #DataNode < # (Blocks in a BlockGroup)

HDFS-8031 Erasure Coding: Sort located striped blocks based on decommissioned states

HDFS-8031 shouldProcessOverReplicated should not count number of pending replicas

HDFS-8031 Missing block exception should carry locatedBlocks information

HDFS-8031 BlockManager#chooseExcessReplicasStriped may weaken rack fault tolerance

HDFS-8031 BlockManager#countNodes should be able to detect duplicated internal blocks

HDFS-8031 Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data

HDFS-8031 Erasure Coding: Improve exception handling in ErasureCodingWorker#ReconstructAndTransferBlock

HDFS-8031 Erasure Coding: Improve few exception handling logic of ErasureCodingWorker

HDFS-8031 Erasure Coding: Postpone the recovery work for a configurable time period

HDFS-8031 Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped block at the same time

HDFS-8031 Correctly handle EC reconstruction work caused by not enough racks

HDFS-8031 Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]

HDFS-8031 Streamer threads may leak if failure happens when closing the striped outputstream

HDFS-8031 Correctly update DataNode's scheduled block size when writing small EC file

HDFS-8031 Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]

HDFS-8031 Refactoring ErasureCodingWorker into smaller reusable constructs

HDFS-8031 Make existing DFSClient#getFileChecksum() work for striped blocks

HDFS-8031 Erasure Coding: allow to use multiple EC policies in striping related tests

HDFS-8031 ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block

HDFS-8031 getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory

HDFS-8031 Move ErasureCodingPolicyManager to FSDirectory

HDFS-8031 Erasure Coding: Wrong limit setting of target ByteBuffer

HDFS-8031 Use byte array for internal block indices in a striped block

HDFS-8031 Erasure coding: an erasure codec throughput benchmark tool

HDFS-8031 Parallel optimization of DFSStripedOutputStream#flushAllInternals( )

HDFS-8031 Erasure coding: some EC tests are missing timeout

HDFS-8031 Erasure coding: updateBlockForPipeline sometimes returns non-striped block for striped file

HDFS-8031 Erasure coding: friendly log information for write operations with some failed streamers

HDFS-8031 Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent file should be handled properly

HDFS-8031 Erasure coding: client should update and commit block based on acknowledged size

HDFS-8031 Wait previous ErasureCodingWork to finish before schedule another one

HDFS-8031 Erasure Coding: Skip encoding the data cells if all the parity data streamers are failed for the current block group

HDFS-8031 Erasure coding: Add apache license header in TestFileStatusWithECPolicy.java

HDFS-8031 Inconsistent default value of dfs.datanode.stripedread.buffer.size

HDFS-8031 Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

HDFS-8031 Erasure Coding: Lease recovery for striped file

HDFS-8031 Erasure Coding: Move DFSStripedIO stream related classes to hadoop-hdfs-client

HDFS-8031 Erasure coding: a comprehensive I/O throughput benchmark tool

HDFS-8031 Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message

HDFS-8031 Refactor DFSInputStream#ReaderStrategy

HDFS-8031 Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping recovery on DataNode side

HDFS-8031 Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping write

HDFS-8031 Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping read (position and stateful)

HDFS-8031 Use ByteBuffer in striping positional read

HDFS-8031 Erasure Coding: cover more test situations of datanode failure during client writing

HDFS-8031 Erasure coding: merge HDFS-8499 to EC branch and refactor BlockInfoStriped

HDFS-8031 Erasure coding: use simple replication for internal blocks on decommissioning datanodes

HDFS-8031 Erasure Coding: add tests for taking snapshots on EC files

HDFS-8031 Erasure Coding: block group ID displayed in WebUI is not consistent with fsck

HDFS-8031 Erasure Coding: the log of each streamer should show its index

HDFS-8031 Erasure Coding: client fails to write large file when one datanode fails

HDFS-8031 Erasure Coding: Add EC-related Metrics to NN (seperate striped blocks count from UnderReplicatedBlocks count)

HDFS-8031 Erasure Coding: revisit buffer used for encoding and decoding.

HDFS-8031 Erasure Coding: use thread pool for StripedDataStreamer

HDFS-8031 Add InterfaceAudience annotation to the erasure coding classes

HDFS-8031 Erasure Coding: update invalidateBlock(..) logic for striped block

HDFS-8031 Erasure Coding: cache ErasureCodingZone

HDFS-8031 Remove the use of hard-coded cell size value in balancer Dispatcher

HDFS-8031 Add blocks count metrics to datanode for ECWorker

HDFS-8031 Erasure Coding: optimize client writing by making the writing of data and parity concurrently

HDFS-8031 Remove hard-coded chunk size in favor of ECZone

HDFS-8031 Add tasks count metrics to datanode for ECWorker

HDFS-8031 Erasure Coding: Allow concat striped files if they have the same ErasureCodingPolicy

HDFS-8031 Add bytes count metrics to datanode for ECWorker

HDFS-8031 Add computation time metrics to datanode for ECWorker

HDFS-8031 Erasure Coding: Correctly calculate last striped block length in DFSStripedInputStream if it's under construction.

HDFS-8031 Erasure Coding: Revisit the long and int datatypes usage in striping logic

HDFS-8031 Tolerate multiple failures in DFSStripedOutputStream

HDFS-8031 Erasure Coding: Update last cellsize calculation according to whether the erasure codec has chunk boundary

HDFS-8031 Ec files can't be deleted into Trash because of that Trash isn't EC zone.

HDFS-8031 Create EC zone should not need superuser privilege

HDFS-8031 Erasure Coding: Create FileStatus isErasureCoded() method

HDFS-8031 Add MODIFY and REMOVE ECSchema editlog operations

HDFS-8031 Merge HDFS-8227 into EC branch

HDFS-8031 ECSchema supports for offline EditsVisitor over an OEV XML file

HDFS-8031 Erasure Coding: Expose refreshECSchemas command to reload predefined schemas

HDFS-8031 Relax permission checking for EC related operations

HDFS-8031 ECManager should be able to manage multiple ECSchemas

HDFS-8031 Allow to configure the system default EC schema

HDFS-8031 Erasure Coding: Add more EC zone management APIs (get/list EC zone(s))

HDFS-8031 Remove hard-coded values in favor of EC schema

HDFS-8031 Erasure coding: revisit how to store EC schema and cellSize in NameNode

HDFS-8031 Erasure Coding: local and remote block writer for coding work in DataNode

HDFS-8031 Erasure Coding: local and remote block reader for coding work in DataNode

HDFS-8031 Erasure coding: NameNode manages multiple erasure coding policies

HDFS-8031 Erasure Coding: Persist erasure coding policies in NameNode

HDFS-8031 Handle hflush and hsync in the best optimal way possible during online Erasure encoding

HDFS-8031 Erasure Coding: unifying common constructs like coding work, block reader and block writer across client and DataNode

HDFS-8031 [umbrella] Adding metrics for Erasure Coding

HDFS-8031 Block Readers and Writers used in both client side and datanode side

HDFS-8031 WebHDFS: Support EC commands through webhdfs

HDFS-8031 [umbrella] Erasure Coding worker and support in DataNode