Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-265

Revisit append

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.

        Attachments

        1. appendDesign3.pdf
          869 kB
          Hairong Kuang
        2. AppendTestPlan.html
          64 kB
          Konstantin I Boudnik
        3. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        4. a.sh
          1 kB
          Tsz-wo Sze
        5. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        6. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        7. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        8. AppendTestPlan.html
          63 kB
          Konstantin I Boudnik
        9. appendDesign2.pdf
          868 kB
          Hairong Kuang
        10. AppendTestPlan.html
          62 kB
          Konstantin I Boudnik
        11. AppendTestPlan.html
          62 kB
          Konstantin I Boudnik
        12. AppendTestPlan.html
          55 kB
          Konstantin I Boudnik
        13. appendDesign1.pdf
          624 kB
          Hairong Kuang
        14. TestPlanAppend.html
          50 kB
          Konstantin I Boudnik
        15. appendDesign.pdf
          408 kB
          Konstantin Shvachko
        16. appendDesign.pdf
          580 kB
          Hairong Kuang
        17. AppendSpec.pdf
          48 kB
          Hairong Kuang

          Issue Links

          1.
          Factor out BlockInfo from BlocksMap Sub-task Closed Konstantin Shvachko
          2.
          Redesign DataNode volumeMap to include all types of Replicas Sub-task Resolved Hairong Kuang
          3.
          Introduce BlockInfoUnderConstruction to reflect block replica states while writing. Sub-task Resolved Konstantin Shvachko
          4.
          Create new tests for Append's hflush Sub-task Resolved Konstantin I Boudnik
          5.
          Create new tests for lease recovery Sub-task Closed Konstantin I Boudnik
          6.
          Create new tests for block recovery Sub-task Closed Hairong Kuang
          7.
          Create new tests for pipeline Sub-task Closed Konstantin I Boudnik
          8.
          Create stress tests for append feature Sub-task Resolved Unassigned
          9.
          Support hflush at DFSClient Sub-task Resolved Hairong Kuang
          10.
          DataNode uses ReplicaBeingWritten to support dfs writes/hflush Sub-task Resolved Hairong Kuang
          11.
          Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append Sub-task Resolved Hairong Kuang
          12.
          Add a "rbw" sub directory to DataNode data directory Sub-task Resolved Hairong Kuang
          13.
          DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas Sub-task Resolved Hairong Kuang
          14.
          Add a test for NameNode.getBlockLocations(..) to check read from un-closed file. Sub-task Resolved Tsz-wo Sze
          15.
          Introduce block committing logic during new block allocation and file close. Sub-task Resolved Konstantin Shvachko
          16.
          When opening a file for read, make the file length avaliable to client. Sub-task Resolved Tsz-wo Sze
          17.
          Extend Block report to include under-construction replicas Sub-task Resolved Konstantin Shvachko
          18.
          Datanode should serve up to visible length of a replica for read requests Sub-task Resolved Tsz-wo Sze
          19.
          Change block write protocol to support pipeline recovery Sub-task Resolved Hairong Kuang
          20.
          Create new functional test for a block report. Sub-task Closed Konstantin I Boudnik
          21.
          Allow client to get a new generation stamp from NameNode Sub-task Resolved Hairong Kuang
          22.
          Block report processing for append Sub-task Resolved Konstantin Shvachko
          23.
          Create functional tests for new design of the block report Sub-task Closed Konstantin I Boudnik
          24.
          Support replica recovery initialization in datanode Sub-task Resolved Tsz-wo Sze
          25.
          Client support pipeline recovery Sub-task Resolved Hairong Kuang
          26.
          Support replica update in datanode Sub-task Resolved Tsz-wo Sze
          27.
          SafeMode should count only complete blocks. Sub-task Resolved Konstantin Shvachko
          28.
          Support pipeline close and close recovery Sub-task Resolved Hairong Kuang
          29.
          Lease recovery, concurrency support. Sub-task Resolved Konstantin Shvachko
          30.
          Replace BlockInfo.isUnderConstruction() with isComplete() Sub-task Resolved Konstantin Shvachko
          31.
          Simplify the codes in the replica related classes Sub-task Resolved Unassigned
          32.
          Remove unused legacy protocol methods. Sub-task Resolved Konstantin Shvachko
          33.
          Block recovery for primary data-node Sub-task Resolved Konstantin Shvachko
          34.
          DFSClient cannot read all the available bytes Sub-task Resolved Tsz-wo Sze
          35.
          Remove deprecated methods from InterDatanodeProtocol. Sub-task Closed Konstantin Shvachko
          36.
          Data-node upgrade problem Sub-task Resolved Hairong Kuang
          37.
          Unnecessary info message from DFSClient Sub-task Resolved Hairong Kuang
          38.
          DFSIO for append Sub-task Closed Konstantin Shvachko
          39.
          TestFileAppend2 sometimes hangs Sub-task Resolved Hairong Kuang
          40.
          TestFileAppend3#TC7 sometimes hangs Sub-task Closed Hairong Kuang
          41.
          NPE in FSDataset.updateReplicaUnderRecovery(..) Sub-task Closed Konstantin Shvachko
          42.
          Create block recovery tests that handle errors Sub-task Closed Hairong Kuang

            Activity

              People

              • Assignee:
                hairong Hairong Kuang
                Reporter:
                hairong Hairong Kuang
              • Votes:
                5 Vote for this issue
                Watchers:
                55 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: