Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5535

Umbrella jira for improved HDFS rolling upgrades

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0, 3.0.0-alpha1
    • Fix Version/s: 2.4.0
    • Component/s: datanode, ha, hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In order to roll a new HDFS release through a large cluster quickly and safely, a few enhancements are needed in HDFS. An initial High level design document will be attached to this jira, and sub-jiras will itemize the individual tasks.

        Attachments

        1. h5535_20140219.patch
          435 kB
          Tsz Wo Nicholas Sze
        2. h5535_20140220-1554.patch
          438 kB
          Tsz Wo Nicholas Sze
        3. h5535_20140220b.patch
          455 kB
          Tsz Wo Nicholas Sze
        4. h5535_20140221-2031.patch
          435 kB
          Tsz Wo Nicholas Sze
        5. h5535_20140224-1931.patch
          495 kB
          Tsz Wo Nicholas Sze
        6. h5535_20140225-1225.patch
          531 kB
          Tsz Wo Nicholas Sze
        7. h5535_20140226-1328.patch
          561 kB
          Tsz Wo Nicholas Sze
        8. h5535_20140226-1911.patch
          561 kB
          Tsz Wo Nicholas Sze
        9. h5535_20140227-1239.patch
          566 kB
          Tsz Wo Nicholas Sze
        10. h5535_20140228-1714.patch
          572 kB
          Tsz Wo Nicholas Sze
        11. h5535_20140304-1138.patch
          579 kB
          Tsz Wo Nicholas Sze
        12. h5535_20140304-branch-2.patch
          601 kB
          Tsz Wo Nicholas Sze
        13. h5535_20140310-branch-2.patch
          602 kB
          Tsz Wo Nicholas Sze
        14. hdfs-5535-test-plan.pdf
          72 kB
          Haohui Mai
        15. HDFSRollingUpgradesHighLevelDesign.pdf
          294 kB
          Suresh Srinivas
        16. HDFSRollingUpgradesHighLevelDesign.v2.pdf
          83 kB
          Suresh Srinivas
        17. HDFSRollingUpgradesHighLevelDesign.v3.pdf
          331 kB
          Kihwal Lee

          Issue Links

          1.
          Revist upgrade snapshots, roll back, finalize to enable rolling upgrades Sub-task Resolved Sanjay Radia
          2.
          Make replication queue initialization asynchronous Sub-task Resolved Vinayakumar B
          3.
          Improve datanode startup time Sub-task Resolved Kihwal Lee
          4.
          Consider supporting a mechanism to allow datanodes to drain outstanding work during rolling upgrade Sub-task Resolved Unassigned
          5.
          Make DN send an OOB Ack on shutdown before restarting Sub-task Resolved Kihwal Lee
          6.
          Add OOB upgrade response and client-side logic for reads Sub-task Resolved Unassigned
          7.
          Provide admin commands for data node upgrade Sub-task Resolved Kihwal Lee
          8.
          Add quick-restart option for datanode Sub-task Resolved Kihwal Lee
          9.
          Support upgrade marker in editlog streams Sub-task Resolved Tsz Wo Nicholas Sze
          10.
          Add a new DFSAdminCommand for rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
          11.
          Support query and finalize rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
          12.
          Add new NN startup options for downgrade and rollback using upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
          13.
          Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion Sub-task Resolved Brandon Li
          14.
          Document new commands and parameters for improved rolling upgrades Sub-task Resolved Tsz Wo Nicholas Sze
          15.
          Add a new option for starting standby NN when rolling upgrade is in progress Sub-task Resolved Tsz Wo Nicholas Sze
          16.
          Add rolling upgrade infomation to heartbeat response Sub-task Resolved Tsz Wo Nicholas Sze
          17.
          Reading edits should not stop at UpgradeMarker for normal restart of the namenode Sub-task Resolved Vinayakumar B
          18.
          When rolling upgrade is in progress, NN should only create checkpoint right before the upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
          19.
          Should not compare DataNode current layout version with that of NameNode in DataStrorage Sub-task Resolved Brandon Li
          20.
          When rolling upgrade is in progress, standby NN should create checkpoint for downgrade. Sub-task Resolved Tsz Wo Nicholas Sze
          21.
          Avoid NPE in Datanode heartbeat Sub-task Resolved Vinayakumar B
          22.
          Upgrade and rolling upgrade should not be allowed simultaneously Sub-task Resolved Tsz Wo Nicholas Sze
          23.
          BlockPoolSliceStorage trash to handle block deletions during rolling upgrade Sub-task Resolved Arpit Agarwal
          24.
          Support rollback of rolling upgrade in NameNode and JournalNodes Sub-task Resolved Jing Zhao
          25.
          Utilize OOB upgrade message processing for writes Sub-task Resolved Kihwal Lee
          26.
          Add rolling upgrade information to fsimage Sub-task Resolved Tsz Wo Nicholas Sze
          27.
          Merge Protobuf-based-FSImage code from trunk Sub-task Resolved Jing Zhao
          28.
          Fix TestRollingUpgrade Sub-task Resolved Arpit Agarwal
          29.
          TestRollingUpgrade#testSecondaryNameNode causes subsequent tests to fail Sub-task Resolved Tsz Wo Nicholas Sze
          30.
          Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
          31.
          Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
          32.
          Fix compilation error after merge Sub-task Resolved Tsz Wo Nicholas Sze
          33.
          Create unit tests for downgrade and finalize Sub-task Resolved Haohui Mai
          34.
          Rollback does not need to load edits Sub-task Resolved Jing Zhao
          35.
          Fix TestEditLog and TestStandbyCheckpoints Sub-task Resolved Jing Zhao
          36.
          SimulatedFSDataset#disableAndPurgeTrashStorage should not throw UnsupportedOperationException Sub-task Resolved Jing Zhao
          37.
          Fix findbugs warnings in Rolling Upgrade branch Sub-task Resolved Tsz Wo Nicholas Sze
          38.
          Fix TestOfflineEditsViewer and NPE in MD5FileUtils Sub-task Resolved Tsz Wo Nicholas Sze
          39.
          Fix TestDataNodeRollingUpgrade failure Sub-task Resolved Arpit Agarwal
          40.
          Do not create rollback fsimage when it already exists Sub-task Resolved Jing Zhao
          41.
          Avoid saving namespace when starting rolling upgrade Sub-task Resolved Jing Zhao
          42.
          Add '-rollingUpgrade <rollback|downgrade>' to namenode usage message Sub-task Resolved Vinayakumar B
          43.
          Change DFSAdmin for rolling upgrade commands Sub-task Resolved Tsz Wo Nicholas Sze
          44.
          Simplify Datanode rollback and downgrade Sub-task Resolved Suresh Srinivas
          45.
          add rollingUpgrade information to latest UI Sub-task Resolved Vinayakumar B
          46.
          Fix findbug warnings introduced by HDFS-5583 Sub-task Resolved Kihwal Lee
          47.
          Fix TestBlockRecovery#testRaceBetweenReplicaRecoveryAndFinalizeBlock Sub-task Resolved Kihwal Lee
          48.
          Update datanode replacement policy to make writes more robust Sub-task Resolved Kihwal Lee
          49.
          Query the status of rolling upgrade in the preparation stage in TestRollingUpgrade and TestRollingUpgradeRollback Sub-task Resolved Haohui Mai
          50.
          Standby NN might not checkpoint when processing the rolling upgrade marker Sub-task Resolved Haohui Mai
          51.
          Fix the five findbugs warnings Sub-task Resolved Kihwal Lee
          52.
          Test whether the standby NN continues to checkpoint after the prepare stage Sub-task Resolved Haohui Mai
          53.
          Test whether the NN will reject the downgrade if it has a fsimage from a newer release Sub-task Resolved Haohui Mai
          54.
          Fix TestDFSUpgrade and TestDataNodeRollingUpgrade Sub-task Resolved Jing Zhao
          55.
          Secondary NN fails to checkpoint after -rollingUpgrade prepare Sub-task Resolved Jing Zhao
          56.
          NN running newer software rejects loading the fsimage during rolling upgrade. Sub-task Resolved Tsz Wo Nicholas Sze
          57.
          -rollingUpgrade query hits NPE after the NN restarts Sub-task Resolved Haohui Mai
          58.
          DN registration should use DataNodeLayoutVersion instead of NameNodeLayoutVersion Sub-task Resolved Tsz Wo Nicholas Sze
          59.
          Allow JournalNode to handle editlog produced by new release with future layoutversion Sub-task Closed Jing Zhao
          60.
          Downgrade/Finalize should rename the rollback image instead of purging it Sub-task Resolved Jing Zhao
          61.
          Fix rolling upgrade documentation and error messges Sub-task Resolved Tsz Wo Nicholas Sze
          62.
          NameNode should not check DataNode layout version Sub-task Closed Brandon Li
          63.
          SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type Sub-task Closed Tsz Wo Nicholas Sze

            Activity

              People

              • Assignee:
                szetszwo Tsz Wo Nicholas Sze
                Reporter:
                nroberts Nathan Roberts
              • Votes:
                1 Vote for this issue
                Watchers:
                36 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: