[HDFS-7285] Erasure Coding Support inside HDFS - ASF JIRA

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.0.0-alpha1
Component/s: None
Labels:
None

Target Version/s:
Hadoop Flags:

Reviewed
Release Note:

Hide

HDFS now provides native support for erasure coding (EC) to store data more efficiently. Each individual directory can be configured with an EC policy with command `hdfs erasurecode -setPolicy`. When a file is created, it will inherit the EC policy from its nearest ancestor directory to determine how its blocks are stored. Compared to 3-way replication, the default EC policy saves 50% of storage space while also tolerating more storage failures.

To support small files, the currently phase of HDFS-EC stores blocks in _striped_ layout, where a logical file block is divided into small units (64KB by default) and distributed to a set of DataNodes. This enables parallel I/O but also decreases data locality. Therefore, the cluster environment and I/O workloads should be considered before configuring EC policies.

Show
 HDFS now provides native support for erasure coding (EC) to store data more efficiently. Each individual directory can be configured with an EC policy with command `hdfs erasurecode -setPolicy`. When a file is created, it will inherit the EC policy from its nearest ancestor directory to determine how its blocks are stored. Compared to 3-way replication, the default EC policy saves 50% of storage space while also tolerating more storage failures. To support small files, the currently phase of HDFS-EC stores blocks in _striped_ layout, where a logical file block is divided into small units (64KB by default) and distributed to a set of DataNodes. This enables parallel I/O but also decreases data locality. Therefore, the cluster environment and I/O workloads should be considered before configuring EC policies.

Description

Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice of data reliability, comparing to the existing HDFS 3-replica approach. For example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, with storage overhead only being 40%. This makes EC a quite attractive alternative for big data storage, particularly for cold data.

Facebook had a related open source project called HDFS-RAID. It used to be one of the contribute packages in HDFS but had been removed since Hadoop 2.0 for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends on MapReduce to do encoding and decoding tasks; 2) it can only be used for cold files that are intended not to be appended anymore; 3) the pure Java EC coding implementation is extremely slow in practical use. Due to these, it might not be a good idea to just bring HDFS-RAID back.

We (Intel and Cloudera) are working on a design to build EC into HDFS that gets rid of any external dependencies, makes it self-contained and independently maintained. This design lays the EC feature on the storage type support and considers compatible with existing HDFS features like caching, snapshot, encryption, high availability and etc. This design will also support different EC coding schemes, implementations and policies for different deployment scenarios. By utilizing advanced libraries (e.g. Intel ISA-L library), an implementation can greatly improve the performance of EC encoding/decoding and makes the EC solution even more attractive. We will post the design document soon.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

1619363340018.png
29/Apr/21 06:19
76 kB
Tsz-wo Sze
Compare-consolidated-20150824.diff
24/Aug/15 18:51
72 kB
Zhe Zhang
Consolidated-20150707.patch
07/Jul/15 23:44
1.02 MB
Zhe Zhang
Consolidated-20150806.patch
07/Aug/15 06:38
1.24 MB
Zhe Zhang
Consolidated-20150810.patch
10/Aug/15 19:49
1.23 MB
Zhe Zhang
ECAnalyzer.py
16/Jan/15 19:29
2 kB
Zhe Zhang
ECParser.py
16/Jan/15 19:29
5 kB
Zhe Zhang
fsimage-analysis-20150105.pdf
05/Jan/15 19:58
82 kB
Zhe Zhang
HDFS-7285-Consolidated-20150911.patch
11/Sep/15 21:25
1.20 MB
Zhe Zhang
HDFS-7285-initial-PoC.patch
06/Mar/15 22:18
470 kB
Zhe Zhang
HDFS-7285-merge-consolidated.trunk.03.patch
02/Jul/15 11:00
1.04 MB
Vinayakumar B
HDFS-7285-merge-consolidated.trunk.04.patch
04/Jul/15 12:21
1.03 MB
Vinayakumar B
HDFS-7285-merge-consolidated-01.patch
01/Jul/15 12:12
1.06 MB
Vinayakumar B
HDFS-7285-merge-consolidated-trunk-01.patch
01/Jul/15 17:05
1.06 MB
Vinayakumar B
HDFS-bistriped.patch
19/Jun/15 19:25
19 kB
Zhe Zhang
HDFS-EC-merge-consolidated-01.patch
01/Jul/15 23:30
1.06 MB
Zhe Zhang
HDFS-EC-Merge-PoC-20150624.patch
24/Jun/15 22:59
811 kB
Zhe Zhang
HDFSErasureCodingDesign-20141028.pdf
29/Oct/14 03:04
1.98 MB
Zhe Zhang
HDFSErasureCodingDesign-20141217.pdf
17/Dec/14 19:51
1.59 MB
Zhe Zhang
HDFSErasureCodingDesign-20150204.pdf
04/Feb/15 21:27
1.40 MB
Tsz-wo Sze
HDFSErasureCodingDesign-20150206.pdf
07/Feb/15 02:04
1.42 MB
Tsz-wo Sze
HDFSErasureCodingPhaseITestPlan.pdf
09/Jun/15 05:33
111 kB
Zhe Zhang
HDFSErasureCodingSystemTestPlan-20150824.pdf
24/Aug/15 02:18
72 kB
Rui Gao
HDFSErasureCodingSystemTestReport-20150826.pdf
28/Aug/15 07:53
218 kB
Rui Gao

Issue Links

incorporates

HADOOP-11264 Common side changes for HDFS Erasure coding support

Resolved

is depended upon by

HDFS-8030 HDFS Erasure Coding Phase II -- EC with contiguous layout

In Progress

HDFS-8031 Follow-on work for erasure coding phase I (striping layout)

Open

is related to

HBASE-19954 Separate TestBlockReorder into individual tests to avoid ShutdownHook suppression error against hadoop3

Resolved

HDFS-503 Implement erasure coding as a layer on HDFS

Closed

HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages

Closed

HDFS-6584 Support Archival Storage

Closed

HDFS-7343 HDFS smart storage management

Open

relates to

HADOOP-12633 Extend Erasure Code to support POWER Chip acceleration

Open

supercedes

MAPREDUCE-3868 Reenable Raid

Resolved

(3 is related to, 1 relates to, 1 supercedes)

Sub-Tasks

1.	Erasure coding: distribute recovery work for striped blocks to DataNode	Resolved	Zhe Zhang
2.	Configurable erasure coding policy for individual files and directories	Resolved	Zhe Zhang
3.	Representing striped block groups in NameNode with hierarchical naming protocol	Resolved	Zhe Zhang
4.	Process block reports for erasure coded blocks	Resolved	Zhe Zhang
5.	[umbrella] Data striping support in HDFS client	Resolved	Li Bo
6.	Document the HDFS Erasure Coding feature	Resolved	Uma Maheswara Rao G
7.	Erasure Coding: extend BlockInfo to handle EC info	Resolved	Jing Zhao
8.	Implement COMPLETE state of erasure coding block groups	Resolved	Zhe Zhang
9.	Add a test for BlockGroup support in FSImage	Resolved	Takuya Fukudome
10.	Add unit tests for editlog transactions for EC	Resolved	Hui Zheng
11.	Change disk quota calculation for EC files	Resolved	Tsz-wo Sze
12.	Erasure Coding: update the Balancer/Mover data migration logic	Resolved	Walter Su
13.	Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks	Resolved	Tsz-wo Sze
14.	Erasure coding: DFSInputStream with decode functionality (pread)	Resolved	Zhe Zhang
15.	Change fsck to support EC files	Resolved	Takanobu Asanuma
16.	Client side api/config changes to support online encoding	Resolved	Vinayakumar B
17.	Add periodic checker to find the corrupted EC blocks/files	Resolved	Vinayakumar B
18.	Avoid Block movement in Balancer and Mover for the erasure encoded blocks	Resolved	Vinayakumar B
19.	Add logic to DFSOutputStream to support writing a file in striping layout	Resolved	Li Bo
20.	Erasure Coding: Add striped block support in INodeFile	Resolved	Jing Zhao
21.	Erasure coding: pread from files in striped layout	Resolved	Zhe Zhang
22.	Support appending to a striping layout file	Resolved	Li Bo
23.	Erasure Coding: Update INodeFile quota computation for striped blocks	Resolved	Kai Sasaki
24.	Erasure Coding: allocate and persist striped blocks in NameNode	Resolved	Jing Zhao
25.	Erasure Coding: support striped blocks in non-protobuf fsimage	Resolved	Hui Zheng
26.	Erasure coding: implement facilities in NameNode to create and manage EC zones	Resolved	Zhe Zhang
27.	Erasure coding: extend LocatedBlocks to support reading from striped files	Resolved	Jing Zhao
28.	Erasure Coding: Update safemode calculation for striped blocks	Resolved	Rui Gao
29.	Erasure Coding: INodeFile.dumpTreeRecursively() supports to print striped blocks	Resolved	Takuya Fukudome
30.	Subclass DFSOutputStream to support writing striping layout files	Resolved	Li Bo
31.	Erasure Coding: track invalid, corrupt, and under-recovery striped blocks in NameNode	Resolved	Jing Zhao
32.	Erasure coding: use BlockInfo[] for both striped and contiguous blocks in INodeFile	Resolved	Zhe Zhang
33.	Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks	Resolved	Jing Zhao
34.	Erasure coding: resolving conflicts in the branch when merging trunk changes.	Resolved	Zhe Zhang
35.	Erasure Coding: INodeFile quota computation unit tests	Resolved	Kai Sasaki
36.	WebImageViewer need support file size calculation with striped blocks	Resolved	Rakesh Radhakrishnan
37.	Erasure coding: NameNode support for lease recovery of striped block groups	Resolved	Zhe Zhang
38.	Erasure coding: Decommission handle for EC blocks.	Resolved	Yi Liu
39.	Erasure coding: DataNode support for block recovery of striped block groups	Resolved	Yi Liu
40.	getStoragePolicy() regards HOT policy as EC policy	Resolved	Takanobu Asanuma
41.	Erasure Coding: simplify striped block recovery work computation and add tests	Resolved	Jing Zhao
42.	Erasure coding: extend UnderReplicatedBlocks to accurately handle striped blocks	Resolved	Zhe Zhang
43.	Erasure Coding: retrieve eraure coding schema for a file from NameNode	Resolved	Vinayakumar B
44.	Erasure Coding: ECworker frame, basics, bootstraping and configuration	Resolved	Uma Maheswara Rao G
45.	Erasure Coding: Update CHANGES-HDFS-7285.txt with branch commits	Resolved	Vinayakumar B
46.	Erasure coding: stateful (non-positional) read from files in striped layout	Resolved	Zhe Zhang
47.	Define a system-wide default EC schema	Resolved	Kai Zheng
48.	Erasure coding: fix bug in EC zone and symlinks	Resolved	Jing Zhao
49.	Erasure Coding: Add RPC to client-namenode to list all ECSchemas loaded in Namenode.	Resolved	Vinayakumar B
50.	Erasure coding: fix bug in TestFSImage	Resolved	Rakesh Radhakrishnan
51.	Make hard-coded values consistent with the system default schema first before remove them	Resolved	Kai Zheng
52.	Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this operation fails	Resolved	Rakesh Radhakrishnan
53.	Erasure coding: created util class to analyze striped block groups	Resolved	Zhe Zhang
54.	BlockManager treates good blocks in a block group as corrput	Resolved	Li Bo
55.	Erasure Coding: Support specifying ECSchema during creation of ECZone	Resolved	Vinayakumar B
56.	Erasure Coding: Better to move EC related proto messages to a separate erasurecoding proto file	Resolved	Rakesh Radhakrishnan
57.	DFSStripedInputStream fails to read data after one stripe	Resolved	Zhe Zhang
58.	Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding	Resolved	Uma Maheswara Rao G
59.	Client gets and uses EC schema when reads and writes a stripping file	Resolved	Kai Sasaki
60.	Send the EC schema to DataNode via EC encoding/recovering command	Resolved	Uma Maheswara Rao G
61.	Fix the editlog corruption exposed by failed TestAddStripedBlocks	Resolved	Jing Zhao
62.	Protobuf changes for BlockECRecoveryCommand and its fields for making it ready for transfer to DN	Resolved	Uma Maheswara Rao G
63.	Support DFS command for the EC encoding	Resolved	Vinayakumar B
64.	Add/implement necessary APIs even we just have the system default schema	Resolved	Kai Zheng
65.	Detect if resevered EC Block ID is already used	Resolved	Hui Zheng
66.	DFSStripedOutputStream should not create empty blocks	Resolved	Jing Zhao
67.	BlockManager.addBlockCollectionWithCheck should check if the block is a striped block	Resolved	Hui Zheng
68.	Erasure Coding: Keep default schema's name consistent	Resolved	Unassigned
69.	Failure handling: DFSStripedOutputStream continues writing with enough remaining datanodes	Resolved	Li Bo
70.	Erasure Coding: DataNode reconstruct striped blocks	Resolved	Yi Liu
71.	createErasureCodingZone sets retryCache state as false always	Resolved	Uma Maheswara Rao G
72.	Erasure Coding: Improve DFSStripedOutputStream closing of datastreamer threads	Resolved	Rakesh Radhakrishnan
73.	Erasure coding: Make block placement policy for EC file configurable	Resolved	Walter Su
74.	Erasure coding: refactor client-related code to sync with HDFS-8082 and HDFS-8169	Resolved	Zhe Zhang
75.	ClientProtocol#createErasureCodingZone API was wrongly annotated as Idempotent	Resolved	Vinayakumar B
76.	StripedBlockUtil.getInternalBlockLength may have overflow error	Resolved	Tsz-wo Sze
77.	Erasure coding: Fix file quota change when we complete/commit the striped blocks	Resolved	Takuya Fukudome
78.	Improve end to end striping file test to add erasure recovering test	Resolved	Xinwei Qin
79.	Erasure Coding: Seek and other Ops in DFSStripedInputStream.	Resolved	Yi Liu
80.	DistributedFileSystem.createErasureCodingZone should pass schema in FileSystemLinkResolver	Resolved	Tsz-wo Sze
81.	TestDFSStripedOutputStream should use BlockReaderTestUtil to create BlockReader	Resolved	Tsz-wo Sze
82.	Erasure Coding: StripedDataStreamer fails to handle the blocklocations which doesn't satisfy BlockGroupSize	Resolved	Rakesh Radhakrishnan
83.	Should calculate checksum for parity blocks in DFSStripedOutputStream	Resolved	Yi Liu
84.	Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY commands from standbynode if any	Resolved	Vinayakumar B
85.	Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause block id conflicts	Resolved	Jing Zhao
86.	Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary	Resolved	Jing Zhao
87.	Erasure Coding: Create DFSStripedInputStream in DFSClient#open	Resolved	Kai Sasaki
88.	Erasure coding: [bug] should always allocate unique striped block group IDs	Resolved	Zhe Zhang
89.	Erasure Coding: XML based end-to-end test for ECCli commands	Resolved	Rakesh Radhakrishnan
90.	DFSStripedOutputStream.closeThreads releases cellBuffers multiple times	Resolved	Kai Sasaki
91.	Avoid assigning a leading streamer in StripedDataStreamer to tolerate datanode failure	Resolved	Tsz-wo Sze
92.	Erasure Coding: simplify the retry logic in DFSStripedInputStream (stateful read)	Resolved	Jing Zhao
93.	Erasure Coding: Implement batched listing of enrasure coding zones	Resolved	Rakesh Radhakrishnan
94.	Erasure Coding: implement parallel stateful reading for striped layout	Resolved	Jing Zhao
95.	Erasure coding: move striped reading logic to StripedBlockUtil	Resolved	Zhe Zhang
96.	Refactor DFSStripedOutputStream and StripedDataStreamer	Resolved	Tsz-wo Sze
97.	Erasure Coding: add ECSchema to HdfsFileStatus	Resolved	Yong Zhang
98.	Erasure Coding: Fix Findbug warnings present in erasure coding	Resolved	Rakesh Radhakrishnan
99.	Erasure Coding: NameNode may get blocked in waitForLoadingFSImage() when loading editlog	Resolved	Jing Zhao
100.	Erasure Coding: DFSStripedOutputStream#close throws NullPointerException exception in some cases	Resolved	Li Bo
101.	Erasure coding: refactor EC constants to be consistent with HDFS-8249	Resolved	Zhe Zhang
102.	Erasure Coding: support decoding for stateful read	Resolved	Jing Zhao
103.	Erasure coding: consolidate striping-related terminologies	Resolved	Zhe Zhang
104.	Bump GenerationStamp for write faliure in DFSStripedOutputStream	Resolved	Tsz-wo Sze
105.	Add trace info to DFSClient#getErasureCodingZoneInfo(..)	Resolved	Vinayakumar B
106.	Follow-on to update decode for DataNode striped blocks reconstruction	Resolved	Yi Liu
107.	Erasure coding: Rename Striped block recovery to reconstruction to eliminate confusion.	Resolved	Yi Liu
108.	Erasure coding: rename DFSStripedInputStream related test classes	Resolved	Zhe Zhang
109.	Expose some administrative erasure coding operations to HdfsAdmin	Resolved	Uma Maheswara Rao G
110.	Erasure Coding: Badly treated when createBlockOutputStream failed in DataStreamer	Resolved	Unassigned
111.	Erasure Coding: test skip in TestDFSStripedInputStream	Resolved	Walter Su
112.	Erasure Coding: test failed in TestDFSStripedInputStream.testStatefulRead() when use ByteBuffer	Resolved	Walter Su
113.	Erasure Coding: whether to use the same chunkSize in decoding with the value in encoding	Resolved	Unassigned
114.	Erasure Coding: test webhdfs read write stripe file	Resolved	Walter Su
115.	Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction	Resolved	Tsz-wo Sze
116.	Fix FindBugs issues introduced by erasure coding	Resolved	Unassigned
117.	Erasure Coding: DFSStripedInputStream#seekToNewSource	Resolved	Yi Liu
118.	Erasure coding: fix some minor bugs in EC CLI	Resolved	Walter Su
119.	Erasure Coding: Badly treated when short of Datanode in StripedDataStreamer	Resolved	Walter Su
120.	Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream	Resolved	Li Bo
121.	BlockInfoStriped uses EC schema	Resolved	Kai Sasaki
122.	Erasure Coding: DFS opening a non-existent file need to be handled properly	Resolved	Rakesh Radhakrishnan
123.	Erasure Coding: TestRecoverStripedFile#testRecoverOneParityBlock is failing	Resolved	Rakesh Radhakrishnan
124.	Erasure coding: compute storage type quotas for striped files, to be consistent with HDFS-8327	Resolved	Zhe Zhang
125.	Add cellSize as an XAttr to ECZone	Resolved	Vinayakumar B
126.	Erasure Coding: Few improvements for the erasure coding worker	Resolved	Rakesh Radhakrishnan
127.	Fix issues like NPE in TestRecoverStripedFile	Resolved	Kai Zheng
128.	Remove chunkSize and initialize from erasure coder	Resolved	Kai Zheng
129.	NN should consider current EC tasks handling count from DN while assigning new tasks	Resolved	Uma Maheswara Rao G
130.	Erasure Coding: unit test the behaviour of BlockManager recovery work for the deleted blocks	Resolved	Rakesh Radhakrishnan
131.	Revisit and refactor ErasureCodingInfo	Resolved	Vinayakumar B
132.	Erasure Coding: Pread failed to read data starting from not-first stripe	Resolved	Walter Su
133.	Fix the isNeededReplication calculation for Striped block in NN	Resolved	Yi Liu
134.	Erasure Coding: ECZoneManager#getECZoneInfo is not resolving the path properly if zone dir itself is the snapshottable dir	Resolved	Rakesh Radhakrishnan
135.	Remove dataBlockNum and parityBlockNum from BlockInfoStriped	Resolved	Kai Sasaki
136.	Erasure Coding: Fix the NullPointerException when deleting file	Resolved	Yi Liu
137.	set blockToken in LocatedStripedBlock	Resolved	Walter Su
138.	Erasure Coding: make condition check earlier for setReplication	Resolved	Walter Su
139.	Erasure Coding: fix cannot rename a zone dir	Resolved	Walter Su
140.	Erasure Coding: Consolidate erasure coding zone related implementation into a single class	Resolved	Rakesh Radhakrishnan
141.	Erasure coding: properly handle start offset for internal blocks in a block group	Resolved	Zhe Zhang
142.	Erasure Coding: stateful read result doesn't match data occasionally because of flawed test	Resolved	Walter Su
143.	Erasure coding: fix priority level of UnderReplicatedBlocks for striped block	Resolved	Walter Su
144.	Refactor BlockInfoContiguous and fix NPE in TestBlockInfo#testCopyConstructor()	Resolved	Vinayakumar B
145.	2 RPC calls for every file read in DFSClient#open(..) resulting in double Audit log entries	Resolved	Vinayakumar B
146.	createErasureCodingZone should check whether cellSize is available	Resolved	Yong Zhang
147.	Erasure coding: fix striping related logic in FSDirWriteFileOp to sync with HDFS-8421	Resolved	Zhe Zhang
148.	Erasure coding: remove workarounds in client side stripped blocks recovering	Resolved	Zhe Zhang
149.	Erasure coding: test DataNode reporting bad/corrupted blocks which belongs to a striped block.	Resolved	Takanobu Asanuma
150.	Erasure coding: Two contiguous blocks occupy IDs belong to same striped group	Resolved	Walter Su
151.	ErasureCodingWorker fails to do decode work	Resolved	Li Bo
152.	Fix a decoding issue in stripped block recovering in client side	Resolved	Kai Zheng
153.	Restore ECZone info inside FSImageLoader	Resolved	Kai Sasaki
154.	Erasure Coding: processOverReplicatedBlock() handles striped block	Resolved	Walter Su
155.	Erasure Coding: Fix FindBugs Multithreaded correctness Warning	Resolved	Rakesh Radhakrishnan
156.	Erasure Coding: Fix usage of 'createZone'	Resolved	Vinayakumar B
157.	Allow to configure RS and XOR raw coders	Resolved	Kai Zheng
158.	Erasure Coding: fix non-protobuf fsimage for striped blocks	Resolved	Jing Zhao
159.	Erasure Coding: fsck handles file smaller than a full stripe	Resolved	Walter Su
160.	Erasure Coding: SafeMode handles file smaller than a full stripe	Resolved	Walter Su
161.	Fix TestErasureCodingCli test	Resolved	Vinayakumar B
162.	Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature	Resolved	Walter Su
163.	Erasure Coding: Remove dataBlockNum and parityBlockNum from StripedBlockProto	Resolved	Yi Liu
164.	Erasure Coding: fix the copy constructor of BlockInfoStriped and BlockInfoContiguous	Resolved	Vinayakumar B
165.	Erasure Coding: Client can't read(decode) the EC files which have corrupt blocks.	Resolved	Kai Sasaki
166.	Erasure Coding: revisit replica counting for striped blocks	Resolved	Jing Zhao
167.	Erasure Coding: handle missing internal block locations in DFSStripedInputStream	Resolved	Jing Zhao
168.	Erasure Coding: fix some block number calculation for striped block	Resolved	Yi Liu
169.	DFSClient hang up when there are not sufficient DataNodes in EC cluster.	Resolved	Kai Sasaki
170.	Erasure coding: update BlockManager.blockHasEnoughRacks(..) logic for striped block	Resolved	Kai Sasaki
171.	Erasure Coding: client generates too many small packets when writing parity data	Resolved	Li Bo
172.	Erasure coding: revisit and simplify BlockInfoStriped and INodeFile	Resolved	Zhe Zhang
173.	Erasure coding: For a small file missing and under replicated ec-block calculation is incorrect	Resolved	J.Andreina
174.	Erasure Coding: Fail to read a file with corrupted blocks	Resolved	Walter Su
175.	Erasure Coding: fix one cell need two packets	Resolved	Walter Su
176.	Erasure Coding: the number of chunks in packet is not updated when writing parity data	Resolved	Li Bo
177.	Erasure Coding: reuse BlockReader when reading the same block in pread	Resolved	Jing Zhao
178.	Erasure Coding: unit test for SequentialBlockGroupIdGenerator	Resolved	Rakesh Radhakrishnan
179.	Erasure Coding: Correctly handle BlockManager#InvalidateBlocks for striped block	Resolved	Yi Liu
180.	Erasure coding: rename BlockInfoContiguousUC and BlockInfoStripedUC to be consistent with trunk	Resolved	Zhe Zhang
181.	Erasure Coding: fix DFSStripedInputStream/DFSStripedOutputStream re-fetch token when expired	Resolved	Walter Su
182.	Erasure Coding: use DirectBufferPool in DFSStripedInputStream for buffer allocation	Resolved	Jing Zhao
183.	Erasure Coding: Client no need to decode missing parity blocks	Resolved	Walter Su
184.	Erasure Coding: add test for namenode process over replicated striped block	Resolved	Takuya Fukudome
185.	Erasure Coding: Fix NPE when NameNode processes over-replicated striped blocks	Resolved	Walter Su
186.	Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones	Resolved	Zhe Zhang
187.	Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream when the data length is small	Resolved	Tsz-wo Sze
188.	Erasure Coding: client occasionally gets less block locations when some datanodes fail	Resolved	Li Bo
189.	Erasure Coding: Provide ECSchema validation when setting EC policy	Resolved	J.Andreina
190.	Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs	Resolved	Walter Su
191.	Erasure Coding: Fix ArrayIndexOutOfBoundsException in TestWriteStripedFileWithFailure	Resolved	Li Bo
192.	Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy	Resolved	Vinayakumar B
193.	Erasure Coding: use threadpool for EC recovery tasks on DataNode	Resolved	Rakesh Radhakrishnan
194.	Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature	Resolved	Jing Zhao
195.	Erasure coding: do not throw exception when setting replication factor on EC files	Resolved	Rui Gao
196.	Erasure coding : Fix random failure in TestSafeModeWithStripedFile	Resolved	J.Andreina
197.	Erasure coding: fix 2 failed tests of DFSStripedOutputStream	Resolved	Walter Su
198.	Erasure coding: MapReduce job failed when I set the / folder to the EC zone	Resolved	Unassigned
199.	Rename dfs.datanode.stripedread.threshold.millis to dfs.datanode.stripedread.timeout.millis	Resolved	Andrew Wang
200.	Cleanup erasure coding documentation	Resolved	Andrew Wang
201.	Erasure Coding: Provide DistributedFilesystem API to getAllErasureCodingPolicies	Resolved	Rakesh Radhakrishnan
202.	Erasure coding: update EC command "-s" flag to "-p" when specifying policy	Resolved	Zhe Zhang
203.	ErasureCodingWorker#processErasureCodingTasks should not fail to process remaining tasks due to one invalid ECTask	Resolved	Uma Maheswara Rao G
204.	Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance	Resolved	Rui Li
205.	Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)	Resolved	Jing Zhao

Activity

People

Assignee:: Zhe Zhang

Reporter:: Weihua Jiang

Votes:: 4 Vote for this issue

Watchers:: 129 Start watching this issue

Dates

Created:: 24/Oct/14 13:02

Updated:: 29/Apr/21 06:20

Resolved:: 30/Sep/15 15:52