Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5535

Umbrella jira for improved HDFS rolling upgrades

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.2.0
    • Fix Version/s: 2.4.0
    • Component/s: datanode, ha, hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In order to roll a new HDFS release through a large cluster quickly and safely, a few enhancements are needed in HDFS. An initial High level design document will be attached to this jira, and sub-jiras will itemize the individual tasks.

      1. h5535_20140219.patch
        435 kB
        Tsz Wo Nicholas Sze
      2. h5535_20140220-1554.patch
        438 kB
        Tsz Wo Nicholas Sze
      3. h5535_20140220b.patch
        455 kB
        Tsz Wo Nicholas Sze
      4. h5535_20140221-2031.patch
        435 kB
        Tsz Wo Nicholas Sze
      5. h5535_20140224-1931.patch
        495 kB
        Tsz Wo Nicholas Sze
      6. h5535_20140225-1225.patch
        531 kB
        Tsz Wo Nicholas Sze
      7. h5535_20140226-1328.patch
        561 kB
        Tsz Wo Nicholas Sze
      8. h5535_20140226-1911.patch
        561 kB
        Tsz Wo Nicholas Sze
      9. h5535_20140227-1239.patch
        566 kB
        Tsz Wo Nicholas Sze
      10. h5535_20140228-1714.patch
        572 kB
        Tsz Wo Nicholas Sze
      11. h5535_20140304-1138.patch
        579 kB
        Tsz Wo Nicholas Sze
      12. h5535_20140304-branch-2.patch
        601 kB
        Tsz Wo Nicholas Sze
      13. h5535_20140310-branch-2.patch
        602 kB
        Tsz Wo Nicholas Sze
      14. hdfs-5535-test-plan.pdf
        72 kB
        Haohui Mai
      15. HDFSRollingUpgradesHighLevelDesign.pdf
        294 kB
        Suresh Srinivas
      16. HDFSRollingUpgradesHighLevelDesign.v2.pdf
        83 kB
        Suresh Srinivas
      17. HDFSRollingUpgradesHighLevelDesign.v3.pdf
        331 kB
        Kihwal Lee

        Issue Links

        1.
        Revist upgrade snapshots, roll back, finalize to enable rolling upgrades Sub-task Resolved Sanjay Radia
         
        2.
        Make replication queue initialization asynchronous Sub-task Resolved Vinayakumar B
         
        3.
        Improve datanode startup time Sub-task Resolved Kihwal Lee
         
        4.
        Consider supporting a mechanism to allow datanodes to drain outstanding work during rolling upgrade Sub-task Resolved Unassigned
         
        5.
        Make DN send an OOB Ack on shutdown before restarting Sub-task Resolved Kihwal Lee
         
        6.
        Add OOB upgrade response and client-side logic for reads Sub-task Resolved Unassigned
         
        7.
        Provide admin commands for data node upgrade Sub-task Resolved Kihwal Lee
         
        8.
        Add quick-restart option for datanode Sub-task Resolved Kihwal Lee
         
        9.
        Support upgrade marker in editlog streams Sub-task Resolved Tsz Wo Nicholas Sze
         
        10.
        Add a new DFSAdminCommand for rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
         
        11.
        Support query and finalize rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
         
        12.
        Add new NN startup options for downgrade and rollback using upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
         
        13.
        Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion Sub-task Resolved Brandon Li
         
        14.
        Document new commands and parameters for improved rolling upgrades Sub-task Resolved Tsz Wo Nicholas Sze
         
        15.
        Add a new option for starting standby NN when rolling upgrade is in progress Sub-task Resolved Tsz Wo Nicholas Sze
         
        16.
        Add rolling upgrade infomation to heartbeat response Sub-task Resolved Tsz Wo Nicholas Sze
         
        17.
        Reading edits should not stop at UpgradeMarker for normal restart of the namenode Sub-task Resolved Vinayakumar B
         
        18.
        When rolling upgrade is in progress, NN should only create checkpoint right before the upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
         
        19.
        Should not compare DataNode current layout version with that of NameNode in DataStrorage Sub-task Resolved Brandon Li
         
        20.
        When rolling upgrade is in progress, standby NN should create checkpoint for downgrade. Sub-task Resolved Tsz Wo Nicholas Sze
         
        21.
        Avoid NPE in Datanode heartbeat Sub-task Resolved Vinayakumar B
         
        22.
        Upgrade and rolling upgrade should not be allowed simultaneously Sub-task Resolved Tsz Wo Nicholas Sze
         
        23.
        BlockPoolSliceStorage trash to handle block deletions during rolling upgrade Sub-task Resolved Arpit Agarwal
         
        24.
        Support rollback of rolling upgrade in NameNode and JournalNodes Sub-task Resolved Jing Zhao
         
        25.
        Utilize OOB upgrade message processing for writes Sub-task Resolved Kihwal Lee
         
        26.
        Add rolling upgrade information to fsimage Sub-task Resolved Tsz Wo Nicholas Sze
         
        27.
        Merge Protobuf-based-FSImage code from trunk Sub-task Resolved Jing Zhao
         
        28.
        Fix TestRollingUpgrade Sub-task Resolved Arpit Agarwal
         
        29.
        TestRollingUpgrade#testSecondaryNameNode causes subsequent tests to fail Sub-task Resolved Tsz Wo Nicholas Sze
         
        30.
        Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
         
        31.
        Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
         
        32.
        Fix compilation error after merge Sub-task Resolved Tsz Wo Nicholas Sze
         
        33.
        Create unit tests for downgrade and finalize Sub-task Resolved Haohui Mai
         
        34.
        Rollback does not need to load edits Sub-task Resolved Jing Zhao
         
        35.
        Fix TestEditLog and TestStandbyCheckpoints Sub-task Resolved Jing Zhao
         
        36.
        SimulatedFSDataset#disableAndPurgeTrashStorage should not throw UnsupportedOperationException Sub-task Resolved Jing Zhao
         
        37.
        Fix findbugs warnings in Rolling Upgrade branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        38.
        Fix TestOfflineEditsViewer and NPE in MD5FileUtils Sub-task Resolved Tsz Wo Nicholas Sze
         
        39.
        Fix TestDataNodeRollingUpgrade failure Sub-task Resolved Arpit Agarwal
         
        40.
        Do not create rollback fsimage when it already exists Sub-task Resolved Jing Zhao
         
        41.
        Avoid saving namespace when starting rolling upgrade Sub-task Resolved Jing Zhao
         
        42.
        Add '-rollingUpgrade <rollback|downgrade>' to namenode usage message Sub-task Resolved Vinayakumar B
         
        43.
        Change DFSAdmin for rolling upgrade commands Sub-task Resolved Tsz Wo Nicholas Sze
         
        44.
        Simplify Datanode rollback and downgrade Sub-task Resolved Suresh Srinivas
         
        45.
        add rollingUpgrade information to latest UI Sub-task Resolved Vinayakumar B
         
        46.
        Fix findbug warnings introduced by HDFS-5583 Sub-task Resolved Kihwal Lee
         
        47.
        Fix TestBlockRecovery#testRaceBetweenReplicaRecoveryAndFinalizeBlock Sub-task Resolved Kihwal Lee
         
        48.
        Update datanode replacement policy to make writes more robust Sub-task Resolved Kihwal Lee
         
        49.
        Query the status of rolling upgrade in the preparation stage in TestRollingUpgrade and TestRollingUpgradeRollback Sub-task Resolved Haohui Mai
         
        50.
        Standby NN might not checkpoint when processing the rolling upgrade marker Sub-task Resolved Haohui Mai
         
        51.
        Fix the five findbugs warnings Sub-task Resolved Kihwal Lee
         
        52.
        Test whether the standby NN continues to checkpoint after the prepare stage Sub-task Resolved Haohui Mai
         
        53.
        Test whether the NN will reject the downgrade if it has a fsimage from a newer release Sub-task Resolved Haohui Mai
         
        54.
        Fix TestDFSUpgrade and TestDataNodeRollingUpgrade Sub-task Resolved Jing Zhao
         
        55.
        Secondary NN fails to checkpoint after -rollingUpgrade prepare Sub-task Resolved Jing Zhao
         
        56.
        NN running newer software rejects loading the fsimage during rolling upgrade. Sub-task Resolved Tsz Wo Nicholas Sze
         
        57.
        -rollingUpgrade query hits NPE after the NN restarts Sub-task Resolved Haohui Mai
         
        58.
        DN registration should use DataNodeLayoutVersion instead of NameNodeLayoutVersion Sub-task Resolved Tsz Wo Nicholas Sze
         
        59.
        Allow JournalNode to handle editlog produced by new release with future layoutversion Sub-task Closed Jing Zhao
         
        60.
        Downgrade/Finalize should rename the rollback image instead of purging it Sub-task Resolved Jing Zhao
         
        61.
        Fix rolling upgrade documentation and error messges Sub-task Resolved Tsz Wo Nicholas Sze
         
        62.
        NameNode should not check DataNode layout version Sub-task Closed Brandon Li
         
        63.
        SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type Sub-task Closed Tsz Wo Nicholas Sze
         

          Activity

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Nathan Roberts
            • Votes:
              1 Vote for this issue
              Watchers:
              37 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development