Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5535

Umbrella jira for improved HDFS rolling upgrades

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0, 3.0.0-alpha1
    • Fix Version/s: 2.4.0
    • Component/s: datanode, ha, hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In order to roll a new HDFS release through a large cluster quickly and safely, a few enhancements are needed in HDFS. An initial High level design document will be attached to this jira, and sub-jiras will itemize the individual tasks.

        Attachments

        1. HDFSRollingUpgradesHighLevelDesign.pdf
          294 kB
          Suresh Srinivas
        2. h5535_20140219.patch
          435 kB
          Tsz Wo Nicholas Sze
        3. h5535_20140220-1554.patch
          438 kB
          Tsz Wo Nicholas Sze
        4. h5535_20140220b.patch
          455 kB
          Tsz Wo Nicholas Sze
        5. h5535_20140221-2031.patch
          435 kB
          Tsz Wo Nicholas Sze
        6. h5535_20140224-1931.patch
          495 kB
          Tsz Wo Nicholas Sze
        7. h5535_20140225-1225.patch
          531 kB
          Tsz Wo Nicholas Sze
        8. hdfs-5535-test-plan.pdf
          72 kB
          Haohui Mai
        9. h5535_20140226-1328.patch
          561 kB
          Tsz Wo Nicholas Sze
        10. h5535_20140226-1911.patch
          561 kB
          Tsz Wo Nicholas Sze
        11. h5535_20140227-1239.patch
          566 kB
          Tsz Wo Nicholas Sze
        12. h5535_20140228-1714.patch
          572 kB
          Tsz Wo Nicholas Sze
        13. HDFSRollingUpgradesHighLevelDesign.v2.pdf
          83 kB
          Suresh Srinivas
        14. h5535_20140304-1138.patch
          579 kB
          Tsz Wo Nicholas Sze
        15. HDFSRollingUpgradesHighLevelDesign.v3.pdf
          331 kB
          Kihwal Lee
        16. h5535_20140304-branch-2.patch
          601 kB
          Tsz Wo Nicholas Sze
        17. h5535_20140310-branch-2.patch
          602 kB
          Tsz Wo Nicholas Sze

          Issue Links

          1.
          Revist upgrade snapshots, roll back, finalize to enable rolling upgrades Sub-task Resolved Sanjay Radia
          2.
          Make replication queue initialization asynchronous Sub-task Resolved Vinayakumar B
          3.
          Improve datanode startup time Sub-task Resolved Kihwal Lee
          4.
          Consider supporting a mechanism to allow datanodes to drain outstanding work during rolling upgrade Sub-task Resolved Unassigned
          5.
          Make DN send an OOB Ack on shutdown before restarting Sub-task Resolved Kihwal Lee
          6.
          Add OOB upgrade response and client-side logic for reads Sub-task Resolved Unassigned
          7.
          Provide admin commands for data node upgrade Sub-task Resolved Kihwal Lee
          8.
          Add quick-restart option for datanode Sub-task Resolved Kihwal Lee
          9.
          Support upgrade marker in editlog streams Sub-task Resolved Tsz Wo Nicholas Sze
          10.
          Add a new DFSAdminCommand for rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
          11.
          Support query and finalize rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
          12.
          Add new NN startup options for downgrade and rollback using upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
          13.
          Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion Sub-task Resolved Brandon Li
          14.
          Document new commands and parameters for improved rolling upgrades Sub-task Resolved Tsz Wo Nicholas Sze
          15.
          Add a new option for starting standby NN when rolling upgrade is in progress Sub-task Resolved Tsz Wo Nicholas Sze
          16.
          Add rolling upgrade infomation to heartbeat response Sub-task Resolved Tsz Wo Nicholas Sze
          17.
          Reading edits should not stop at UpgradeMarker for normal restart of the namenode Sub-task Resolved Vinayakumar B
          18.
          When rolling upgrade is in progress, NN should only create checkpoint right before the upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
          19.
          Should not compare DataNode current layout version with that of NameNode in DataStrorage Sub-task Resolved Brandon Li
          20.
          When rolling upgrade is in progress, standby NN should create checkpoint for downgrade. Sub-task Resolved Tsz Wo Nicholas Sze
          21.
          Avoid NPE in Datanode heartbeat Sub-task Resolved Vinayakumar B
          22.
          Upgrade and rolling upgrade should not be allowed simultaneously Sub-task Resolved Tsz Wo Nicholas Sze
          23.
          BlockPoolSliceStorage trash to handle block deletions during rolling upgrade Sub-task Resolved Arpit Agarwal
          24.
          Support rollback of rolling upgrade in NameNode and JournalNodes Sub-task Resolved Jing Zhao
          25.
          Utilize OOB upgrade message processing for writes Sub-task Resolved Kihwal Lee
          26.
          Add rolling upgrade information to fsimage Sub-task Resolved Tsz Wo Nicholas Sze
          27.
          Merge Protobuf-based-FSImage code from trunk Sub-task Resolved Jing Zhao
          28.
          Fix TestRollingUpgrade Sub-task Resolved Arpit Agarwal
          29.
          TestRollingUpgrade#testSecondaryNameNode causes subsequent tests to fail Sub-task Resolved Tsz Wo Nicholas Sze
          30.
          Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
          31.
          Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
          32.
          Fix compilation error after merge Sub-task Resolved Tsz Wo Nicholas Sze
          33.
          Create unit tests for downgrade and finalize Sub-task Resolved Haohui Mai
          34.
          Rollback does not need to load edits Sub-task Resolved Jing Zhao
          35.
          Fix TestEditLog and TestStandbyCheckpoints Sub-task Resolved Jing Zhao
          36.
          SimulatedFSDataset#disableAndPurgeTrashStorage should not throw UnsupportedOperationException Sub-task Resolved Jing Zhao
          37.
          Fix findbugs warnings in Rolling Upgrade branch Sub-task Resolved Tsz Wo Nicholas Sze
          38.
          Fix TestOfflineEditsViewer and NPE in MD5FileUtils Sub-task Resolved Tsz Wo Nicholas Sze
          39.
          Fix TestDataNodeRollingUpgrade failure Sub-task Resolved Arpit Agarwal
          40.
          Do not create rollback fsimage when it already exists Sub-task Resolved Jing Zhao
          41.
          Avoid saving namespace when starting rolling upgrade Sub-task Resolved Jing Zhao
          42.
          Add '-rollingUpgrade <rollback|downgrade>' to namenode usage message Sub-task Resolved Vinayakumar B
          43.
          Change DFSAdmin for rolling upgrade commands Sub-task Resolved Tsz Wo Nicholas Sze
          44.
          Simplify Datanode rollback and downgrade Sub-task Resolved Suresh Srinivas
          45.
          add rollingUpgrade information to latest UI Sub-task Resolved Vinayakumar B
          46.
          Fix findbug warnings introduced by HDFS-5583 Sub-task Resolved Kihwal Lee
          47.
          Fix TestBlockRecovery#testRaceBetweenReplicaRecoveryAndFinalizeBlock Sub-task Resolved Kihwal Lee
          48.
          Update datanode replacement policy to make writes more robust Sub-task Resolved Kihwal Lee
          49.
          Query the status of rolling upgrade in the preparation stage in TestRollingUpgrade and TestRollingUpgradeRollback Sub-task Resolved Haohui Mai
          50.
          Standby NN might not checkpoint when processing the rolling upgrade marker Sub-task Resolved Haohui Mai
          51.
          Fix the five findbugs warnings Sub-task Resolved Kihwal Lee
          52.
          Test whether the standby NN continues to checkpoint after the prepare stage Sub-task Resolved Haohui Mai
          53.
          Test whether the NN will reject the downgrade if it has a fsimage from a newer release Sub-task Resolved Haohui Mai
          54.
          Fix TestDFSUpgrade and TestDataNodeRollingUpgrade Sub-task Resolved Jing Zhao
          55.
          Secondary NN fails to checkpoint after -rollingUpgrade prepare Sub-task Resolved Jing Zhao
          56.
          NN running newer software rejects loading the fsimage during rolling upgrade. Sub-task Resolved Tsz Wo Nicholas Sze
          57.
          -rollingUpgrade query hits NPE after the NN restarts Sub-task Resolved Haohui Mai
          58.
          DN registration should use DataNodeLayoutVersion instead of NameNodeLayoutVersion Sub-task Resolved Tsz Wo Nicholas Sze
          59.
          Allow JournalNode to handle editlog produced by new release with future layoutversion Sub-task Closed Jing Zhao
          60.
          Downgrade/Finalize should rename the rollback image instead of purging it Sub-task Resolved Jing Zhao
          61.
          Fix rolling upgrade documentation and error messges Sub-task Resolved Tsz Wo Nicholas Sze
          62.
          NameNode should not check DataNode layout version Sub-task Closed Brandon Li
          63.
          SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type Sub-task Closed Tsz Wo Nicholas Sze

            Activity

              People

              • Assignee:
                szetszwo Tsz Wo Nicholas Sze
                Reporter:
                nroberts Nathan Roberts
              • Votes:
                1 Vote for this issue
                Watchers:
                36 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: