Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.

      1. appendDesign3.pdf
        869 kB
        Hairong Kuang
      2. AppendTestPlan.html
        64 kB
        Konstantin Boudnik
      3. AppendTestPlan.html
        65 kB
        Konstantin Boudnik
      4. a.sh
        1 kB
        Tsz Wo Nicholas Sze
      5. AppendTestPlan.html
        65 kB
        Konstantin Boudnik
      6. AppendTestPlan.html
        65 kB
        Konstantin Boudnik
      7. AppendTestPlan.html
        65 kB
        Konstantin Boudnik
      8. AppendTestPlan.html
        63 kB
        Konstantin Boudnik
      9. appendDesign2.pdf
        868 kB
        Hairong Kuang
      10. AppendTestPlan.html
        62 kB
        Konstantin Boudnik
      11. AppendTestPlan.html
        62 kB
        Konstantin Boudnik
      12. AppendTestPlan.html
        55 kB
        Konstantin Boudnik
      13. appendDesign1.pdf
        624 kB
        Hairong Kuang
      14. TestPlanAppend.html
        50 kB
        Konstantin Boudnik
      15. appendDesign.pdf
        408 kB
        Konstantin Shvachko
      16. appendDesign.pdf
        580 kB
        Hairong Kuang
      17. AppendSpec.pdf
        48 kB
        Hairong Kuang

        Issue Links

        1.
        Factor out BlockInfo from BlocksMap Sub-task Closed Konstantin Shvachko
         
        2.
        Redesign DataNode volumeMap to include all types of Replicas Sub-task Resolved Hairong Kuang
         
        3.
        Introduce BlockInfoUnderConstruction to reflect block replica states while writing. Sub-task Resolved Konstantin Shvachko
         
        4.
        Create new tests for Append's hflush Sub-task Resolved Konstantin Boudnik
         
        5.
        Create new tests for lease recovery Sub-task Closed Konstantin Boudnik
         
        6.
        Create new tests for block recovery Sub-task Closed Hairong Kuang
         
        7.
        Create new tests for pipeline Sub-task Closed Konstantin Boudnik
         
        8.
        Create stress tests for append feature Sub-task Resolved Unassigned
         
        9.
        Support hflush at DFSClient Sub-task Resolved Hairong Kuang
         
        10.
        DataNode uses ReplicaBeingWritten to support dfs writes/hflush Sub-task Resolved Hairong Kuang
         
        11.
        Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append Sub-task Resolved Hairong Kuang
         
        12.
        Add a "rbw" sub directory to DataNode data directory Sub-task Resolved Hairong Kuang
         
        13.
        DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas Sub-task Resolved Hairong Kuang
         
        14.
        Add a test for NameNode.getBlockLocations(..) to check read from un-closed file. Sub-task Resolved Tsz Wo Nicholas Sze
         
        15.
        Introduce block committing logic during new block allocation and file close. Sub-task Resolved Konstantin Shvachko
         
        16.
        When opening a file for read, make the file length avaliable to client. Sub-task Resolved Tsz Wo Nicholas Sze
         
        17.
        Extend Block report to include under-construction replicas Sub-task Resolved Konstantin Shvachko
         
        18.
        Datanode should serve up to visible length of a replica for read requests Sub-task Resolved Tsz Wo Nicholas Sze
         
        19.
        Change block write protocol to support pipeline recovery Sub-task Resolved Hairong Kuang
         
        20.
        Create new functional test for a block report. Sub-task Closed Konstantin Boudnik
         
        21.
        Allow client to get a new generation stamp from NameNode Sub-task Resolved Hairong Kuang
         
        22.
        Block report processing for append Sub-task Resolved Konstantin Shvachko
         
        23.
        Create functional tests for new design of the block report Sub-task Closed Konstantin Boudnik
         
        24.
        Support replica recovery initialization in datanode Sub-task Resolved Tsz Wo Nicholas Sze
         
        25.
        Client support pipeline recovery Sub-task Resolved Hairong Kuang
         
        26.
        Support replica update in datanode Sub-task Resolved Tsz Wo Nicholas Sze
         
        27.
        SafeMode should count only complete blocks. Sub-task Resolved Konstantin Shvachko
         
        28.
        Support pipeline close and close recovery Sub-task Resolved Hairong Kuang
         
        29.
        Lease recovery, concurrency support. Sub-task Resolved Konstantin Shvachko
         
        30.
        Replace BlockInfo.isUnderConstruction() with isComplete() Sub-task Resolved Konstantin Shvachko
         
        31.
        Simplify the codes in the replica related classes Sub-task Resolved Unassigned
         
        32.
        Remove unused legacy protocol methods. Sub-task Resolved Konstantin Shvachko
         
        33.
        Block recovery for primary data-node Sub-task Resolved Konstantin Shvachko
         
        34.
        DFSClient cannot read all the available bytes Sub-task Resolved Tsz Wo Nicholas Sze
         
        35.
        Remove deprecated methods from InterDatanodeProtocol. Sub-task Closed Konstantin Shvachko
         
        36.
        Data-node upgrade problem Sub-task Resolved Hairong Kuang
         
        37.
        Unnecessary info message from DFSClient Sub-task Resolved Hairong Kuang
         
        38.
        DFSIO for append Sub-task Closed Konstantin Shvachko
         
        39.
        TestFileAppend2 sometimes hangs Sub-task Resolved Hairong Kuang
         
        40.
        TestFileAppend3#TC7 sometimes hangs Sub-task Closed Hairong Kuang
         
        41.
        NPE in FSDataset.updateReplicaUnderRecovery(..) Sub-task Closed Konstantin Shvachko
         
        42.
        Create block recovery tests that handle errors Sub-task Closed Hairong Kuang
         

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              5 Vote for this issue
              Watchers:
              58 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development