Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8031

Follow-on work for erasure coding phase I (striping layout)

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

      Issue Links

      1.
      Block Readers and Writers used in both client side and datanode side Sub-task Resolved Li Bo Actions
      2.
      Erasure Coding: unifying common constructs like coding work, block reader and block writer across client and DataNode Sub-task Resolved Li Bo Actions
      3.
      WebHDFS: Support EC commands through webhdfs Sub-task Resolved Uma Maheswara Rao G Actions
      4.
      Handle hflush and hsync in the best optimal way possible during online Erasure encoding Sub-task Resolved Vinayakumar B Actions
      5.
      Erasure Coding: Add more EC zone management APIs (get/list EC zone(s)) Sub-task Resolved Yi Liu Actions
      6.
      Allow to configure the system default EC schema Sub-task Resolved Kai Zheng Actions
      7.
      ECManager should be able to manage multiple ECSchemas Sub-task Resolved Unassigned Actions
      8.
      Erasure Coding: Persist erasure coding policies in NameNode Sub-task Resolved Sammi Chen Actions
      9.
      Erasure coding: NameNode manages multiple erasure coding policies Sub-task Resolved Rui Li Actions
      10.
      [umbrella] Adding metrics for Erasure Coding Sub-task Resolved Li Bo Actions
      11.
      ECSchema supports for offline EditsVisitor over an OEV XML file Sub-task Resolved Xinwei Qin Actions
      12.
      Erasure Coding: Expose refreshECSchemas command to reload predefined schemas Sub-task Resolved Rakesh Radhakrishnan Actions
      13.
      Erasure coding: revisit how to store EC schema and cellSize in NameNode Sub-task Resolved Yi Liu Actions
      14.
      Add MODIFY and REMOVE ECSchema editlog operations Sub-task Resolved Xinwei Qin Actions
      15.
      Erasure Coding: local and remote block reader for coding work in DataNode Sub-task Resolved Zhe Zhang Actions
      16.
      Erasure Coding: Create FileStatus isErasureCoded() method Sub-task Resolved Rakesh Radhakrishnan Actions
      17.
      Erasure Coding: Update last cellsize calculation according to whether the erasure codec has chunk boundary Sub-task Resolved Yi Liu Actions
      18.
      Erasure Coding: local and remote block writer for coding work in DataNode Sub-task Resolved Li Bo Actions
      19.
      [umbrella] Erasure Coding worker and support in DataNode Sub-task Resolved Li Bo Actions
      20.
      Erasure Coding: Correctly calculate last striped block length in DFSStripedInputStream if it's under construction. Sub-task Resolved Yi Liu Actions
      21.
      Add computation time metrics to datanode for ECWorker Sub-task Resolved Sammi Chen Actions
      22.
      Add bytes count metrics to datanode for ECWorker Sub-task Resolved Sammi Chen Actions
      23.
      Erasure Coding: Allow concat striped files if they have the same ErasureCodingPolicy Sub-task Resolved Walter Su Actions
      24.
      Add tasks count metrics to datanode for ECWorker Sub-task Resolved Li Bo Actions
      25.
      Ec files can't be deleted into Trash because of that Trash isn't EC zone. Sub-task Resolved Brahma Reddy Battula Actions
      26.
      Relax permission checking for EC related operations Sub-task Resolved Andrew Wang Actions
      27.
      Merge HDFS-8227 into EC branch Sub-task Resolved Haohui Mai Actions
      28.
      Create EC zone should not need superuser privilege Sub-task Resolved Yong Zhang Actions
      29.
      Erasure Coding: optimize client writing by making the writing of data and parity concurrently Sub-task Resolved Li Bo Actions
      30.
      Add blocks count metrics to datanode for ECWorker Sub-task Resolved Li Bo Actions
      31.
      Erasure Coding: cache ErasureCodingZone Sub-task Resolved Walter Su Actions
      32.
      Remove hard-coded chunk size in favor of ECZone Sub-task Resolved Kai Sasaki Actions
      33.
      Remove hard-coded values in favor of EC schema Sub-task Resolved Kai Sasaki Actions
      34.
      Erasure Coding: update invalidateBlock(..) logic for striped block Sub-task Resolved Walter Su Actions
      35.
      Erasure Coding: use thread pool for StripedDataStreamer Sub-task Resolved Rui Gao Actions
      36.
      Erasure Coding: revisit buffer used for encoding and decoding. Sub-task Resolved Sammi Chen Actions
      37.
      Erasure Coding: Add EC-related Metrics to NN (seperate striped blocks count from UnderReplicatedBlocks count) Sub-task Resolved Manoj Govindassamy Actions
      38.
      Erasure coding: use simple replication for internal blocks on decommissioning datanodes Sub-task Resolved Rakesh Radhakrishnan Actions
      39.
      Erasure Coding: Revisit the long and int datatypes usage in striping logic Sub-task Resolved Rakesh Radhakrishnan Actions
      40.
      Erasure Coding: block group ID displayed in WebUI is not consistent with fsck Sub-task Resolved Unassigned Actions
      41.
      Remove the use of hard-coded cell size value in balancer Dispatcher Sub-task Resolved Walter Su Actions
      42.
      Erasure coding: merge HDFS-8499 to EC branch and refactor BlockInfoStriped Sub-task Resolved Zhe Zhang Actions
      43.
      Use ByteBuffer in striping positional read Sub-task Resolved Sammi Chen Actions
      44.
      Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping read (position and stateful) Sub-task Resolved Kai Zheng Actions
      45.
      Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping write Sub-task Resolved Kai Zheng Actions
      46.
      Uses ByteBuffer on heap or direct ByteBuffer according to used erasure coder in striping recovery on DataNode side Sub-task Resolved Kai Zheng Actions
      47.
      Refactor DFSInputStream#ReaderStrategy Sub-task Resolved Sammi Chen Actions
      48.
      Erasure Coding: the log of each streamer should show its index Sub-task Resolved Li Bo Actions
      49.
      Erasure coding: a comprehensive I/O throughput benchmark tool Sub-task Resolved Rui Li Actions
      50.
      Tolerate multiple failures in DFSStripedOutputStream Sub-task Resolved Walter Su Actions
      51.
      Erasure Coding: client fails to write large file when one datanode fails Sub-task Resolved Li Bo Actions
      52.
      Erasure Coding: add tests for taking snapshots on EC files Sub-task Resolved Rakesh Radhakrishnan Actions
      53.
      Simplify Erasure Coding Zone DiskSpace quota exceeded exception error message Sub-task Resolved Rui Gao Actions
      54.
      Erasure Coding: cover more test situations of datanode failure during client writing Sub-task Resolved Li Bo Actions
      55.
      Erasure Coding: Move DFSStripedIO stream related classes to hadoop-hdfs-client Sub-task Resolved Zhe Zhang Actions
      56.
      Erasure Coding: Lease recovery for striped file Sub-task Resolved Walter Su Actions
      57.
      Add InterfaceAudience annotation to the erasure coding classes Sub-task Resolved Rakesh Radhakrishnan Actions
      58.
      Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers Sub-task Resolved Jing Zhao Actions
      59.
      Inconsistent default value of dfs.datanode.stripedread.buffer.size Sub-task Resolved Walter Su Actions
      60.
      Erasure coding: Add apache license header in TestFileStatusWithECPolicy.java Sub-task Resolved Surendra Singh Lilhore Actions
      61.
      Erasure Coding: Skip encoding the data cells if all the parity data streamers are failed for the current block group Sub-task Resolved Rakesh Radhakrishnan Actions
      62.
      Wait previous ErasureCodingWork to finish before schedule another one Sub-task Resolved Walter Su Actions
      63.
      Erasure coding: client should update and commit block based on acknowledged size Sub-task Resolved Sammi Chen Actions
      64.
      Erasure coding: friendly log information for write operations with some failed streamers Sub-task Resolved Li Bo Actions
      65.
      Erasure coding: updateBlockForPipeline sometimes returns non-striped block for striped file Sub-task Resolved Unassigned Actions
      66.
      Erasure coding: some EC tests are missing timeout Sub-task Resolved Rui Gao Actions
      67.
      Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent file should be handled properly Sub-task Resolved Rakesh Radhakrishnan Actions
      68.
      Parallel optimization of DFSStripedOutputStream#flushAllInternals( ) Sub-task Resolved Rui Gao Actions
      69.
      Erasure coding: an erasure codec throughput benchmark tool Sub-task Resolved Unassigned Actions
      70.
      Use byte array for internal block indices in a striped block Sub-task Resolved Jing Zhao Actions
      71.
      Erasure Coding: Wrong limit setting of target ByteBuffer Sub-task Resolved Kai Actions
      72.
      Move ErasureCodingPolicyManager to FSDirectory Sub-task Resolved Walter Su Actions
      73.
      getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory Sub-task Resolved Jing Zhao Actions
      74.
      ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block Sub-task Resolved Jing Zhao Actions
      75.
      Erasure Coding: allow to use multiple EC policies in striping related tests Sub-task Resolved Rui Li Actions
      76.
      Make existing DFSClient#getFileChecksum() work for striped blocks Sub-task Resolved Kai Zheng Actions
      77.
      Refactoring ErasureCodingWorker into smaller reusable constructs Sub-task Resolved Kai Zheng Actions
      78.
      Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2] Sub-task Resolved Rui Li Actions
      79.
      Correctly update DataNode's scheduled block size when writing small EC file Sub-task Resolved Jing Zhao Actions
      80.
      Streamer threads may leak if failure happens when closing the striped outputstream Sub-task Resolved Jing Zhao Actions
      81.
      Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3] Sub-task Resolved Rui Li Actions
      82.
      Correctly handle EC reconstruction work caused by not enough racks Sub-task Resolved Jing Zhao Actions
      83.
      Erasure Coding: Postpone the recovery work for a configurable time period Sub-task Resolved Li Bo Actions
      84.
      Erasure Coding: Improve few exception handling logic of ErasureCodingWorker Sub-task Resolved Rakesh Radhakrishnan Actions
      85.
      Erasure Coding: Improve exception handling in ErasureCodingWorker#ReconstructAndTransferBlock Sub-task Resolved Yiqun Lin Actions
      86.
      Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data Sub-task Resolved Rakesh Radhakrishnan Actions
      87.
      BlockManager#countNodes should be able to detect duplicated internal blocks Sub-task Resolved Jing Zhao Actions
      88.
      BlockManager#chooseExcessReplicasStriped may weaken rack fault tolerance Sub-task Resolved Jing Zhao Actions
      89.
      Missing block exception should carry locatedBlocks information Sub-task Resolved Mingliang Liu Actions
      90.
      shouldProcessOverReplicated should not count number of pending replicas Sub-task Resolved Jing Zhao Actions
      91.
      Erasure Coding: Avoids scheduling multiple reconstruction tasks for a striped block at the same time Sub-task Open Sammi Chen Actions
      92.
      Erasure Coding: Sort located striped blocks based on decommissioned states Sub-task Resolved Rakesh Radhakrishnan Actions
      93.
      Erasure Coding: support small cluster whose #DataNode < # (Blocks in a BlockGroup) Sub-task Resolved Li Bo Actions
      94.
      StripedFileTestUtil#readAll flaky Sub-task Resolved Mingliang Liu Actions
      95.
      Fix intermittent test failure of TestDataNodeErasureCodingMetrics Sub-task Resolved Rakesh Radhakrishnan Actions
      96.
      Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block Sub-task Resolved Rakesh Radhakrishnan Actions
      97.
      Allow only suitable storage policies to be set on striped files Sub-task Resolved Uma Maheswara Rao G Actions
      98.
      BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy Sub-task Resolved Manoj Govindassamy Actions
      99.
      Add EC policy and storage policy related usage summarization function to dfs du command Sub-task Resolved Sammi Chen Actions
      100.
      Erasure Coding: Document about the current allowed storage policies for EC Striped mode files Sub-task Resolved Uma Maheswara Rao G Actions
      101.
      Erasure Coding: Add removeErasureCodingPolicy API Sub-task Resolved Unassigned Actions
      102.
      Correctly report missing EC blocks in FSCK Sub-task Resolved Takanobu Asanuma Actions
      103.
      When there are unrecoverable ec block groups, Namenode Web UI shows "There are X missing blocks." but doesn't show the block names. Sub-task Resolved Takanobu Asanuma Actions
      104.
      FBR processing may generate incorrect reportedBlock-blockGroup mapping Sub-task Resolved Jing Zhao Actions
      105.
      Switch from "raw" to "system" xattr namespace for erasure coding policy Sub-task Resolved Andrew Wang Actions
      106.
      BlockManager#isInNewRack should consider decommissioning nodes Sub-task Resolved Jing Zhao Actions
      107.
      Distcp should not copy replication factor if source file is erasure coded Sub-task Resolved Manoj Govindassamy Actions
      108.
      fsck -list-corruptfileblocks does not report corrupt EC files Sub-task Resolved Takanobu Asanuma Actions
      109.
      Report erasure coding policy of EC files in Fsck Sub-task Resolved Wei-Chiu Chuang Actions
      110.
      OIV tool should make an EC file explicit Sub-task Resolved Manoj Govindassamy Actions
      111.
      Ability to specify per-file EC policy at create time Sub-task Resolved Sammi Chen Actions
      112.
      Support an XOR policy XOR-2-1-64k in HDFS Sub-task Resolved Sammi Chen Actions
      113.
      Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats Sub-task Resolved Manoj Govindassamy Actions
      114.
      Correct typos in native erasure coding dump code Sub-task Resolved László Bence Nagy Actions
      115.
      Improve test coverage for ISA-L native coder Sub-task Open Huafeng Wang Actions
      116.
      Add ability to unset and change directory EC policy Sub-task Resolved Sammi Chen Actions
      117.
      Provide replicated EC policy to replicate files Sub-task Resolved Sammi Chen Actions
      118.
      Document dfs.client.read.striped configuration in hdfs-default.xml Sub-task Resolved Rakesh Radhakrishnan Actions
      119.
      Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping Sub-task Resolved Takanobu Asanuma Actions
      120.
      Report blockIds of internal blocks for EC files in Fsck Sub-task Resolved Takanobu Asanuma Actions
      121.
      Support an erasure coding policy using RS 10 + 4 Sub-task Resolved Wei Zhou Actions
      122.
      Enforce set of enabled EC policies on the NameNode Sub-task Resolved Andrew Wang Actions
      123.
      Call RawErasureEncoder and RawErasureDecoder release() methods Sub-task Resolved Sammi Chen Actions
      124.
      Erasure Coding: Support Parity Blocks placement onto same nodes hosting Data Blocks when DataNodes are insufficient Sub-task Resolved Manoj Govindassamy Actions
      125.
      Support ErasureCoding section in OIV XML/ReverseXML Sub-task Resolved Huafeng Wang Actions
      126.
      Add javadoc for storage policy and erasure coding policy Sub-task Resolved Kai Actions
      127.
      Inotify should support erasure coding policy op as replica meta change Sub-task Resolved Huafeng Wang Actions
      128.
      Erasure Coding: Miss EC block after decommission and restart NN Sub-task Open Wu Weiwei Actions
      129.
      Erasure Coding: metrics xmitsInProgress become to negative. Sub-task Resolved Baolong Mao Actions
      130.
      Erasure Coding: decommissioning datanodes cause replicate a large number of duplicate EC internal blocks Sub-task Resolved Wu Weiwei Actions

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            zhz Zhe Zhang

            Dates

            • Created:
              Updated:

              Issue deployment