Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5535

Umbrella jira for improved HDFS rolling upgrades

          Details

          • Type: New Feature
          • Status: Closed
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: 2.2.0, 3.0.0-alpha1
          • Fix Version/s: 2.4.0
          • Component/s: datanode, ha, hdfs-client, namenode
          • Labels:
            None
          • Hadoop Flags:
            Reviewed

            Description

            In order to roll a new HDFS release through a large cluster quickly and safely, a few enhancements are needed in HDFS. An initial High level design document will be attached to this jira, and sub-jiras will itemize the individual tasks.

              Attachments

              1. h5535_20140219.patch
                435 kB
                Tsz Wo Nicholas Sze
              2. h5535_20140220-1554.patch
                438 kB
                Tsz Wo Nicholas Sze
              3. h5535_20140220b.patch
                455 kB
                Tsz Wo Nicholas Sze
              4. h5535_20140221-2031.patch
                435 kB
                Tsz Wo Nicholas Sze
              5. h5535_20140224-1931.patch
                495 kB
                Tsz Wo Nicholas Sze
              6. h5535_20140225-1225.patch
                531 kB
                Tsz Wo Nicholas Sze
              7. h5535_20140226-1328.patch
                561 kB
                Tsz Wo Nicholas Sze
              8. h5535_20140226-1911.patch
                561 kB
                Tsz Wo Nicholas Sze
              9. h5535_20140227-1239.patch
                566 kB
                Tsz Wo Nicholas Sze
              10. h5535_20140228-1714.patch
                572 kB
                Tsz Wo Nicholas Sze
              11. h5535_20140304-1138.patch
                579 kB
                Tsz Wo Nicholas Sze
              12. h5535_20140304-branch-2.patch
                601 kB
                Tsz Wo Nicholas Sze
              13. h5535_20140310-branch-2.patch
                602 kB
                Tsz Wo Nicholas Sze
              14. hdfs-5535-test-plan.pdf
                72 kB
                Haohui Mai
              15. HDFSRollingUpgradesHighLevelDesign.pdf
                294 kB
                Suresh Srinivas
              16. HDFSRollingUpgradesHighLevelDesign.v2.pdf
                83 kB
                Suresh Srinivas
              17. HDFSRollingUpgradesHighLevelDesign.v3.pdf
                331 kB
                Kihwal Lee

                Issue Links

                is related to

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-7230 Add rolling downgrade documentation

                • Major - Major loss of function.
                • Closed
                relates to

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-6800 Support Datanode layout changes with rolling upgrade

                • Major - Major loss of function.
                • Closed
                1.
                		 Revist upgrade snapshots, roll back, finalize to enable rolling upgrades Sub-task Resolved Sanjay Radia
                2.
                		 Make replication queue initialization asynchronous Sub-task Resolved Vinayakumar B
                3.
                		 Improve datanode startup time Sub-task Resolved Kihwal Lee
                4.
                		 Consider supporting a mechanism to allow datanodes to drain outstanding work during rolling upgrade Sub-task Resolved Unassigned
                5.
                		 Make DN send an OOB Ack on shutdown before restarting Sub-task Resolved Kihwal Lee
                6.
                		 Add OOB upgrade response and client-side logic for reads Sub-task Resolved Unassigned
                7.
                		 Provide admin commands for data node upgrade Sub-task Resolved Kihwal Lee
                8.
                		 Add quick-restart option for datanode Sub-task Resolved Kihwal Lee
                9.
                		 Support upgrade marker in editlog streams Sub-task Resolved Tsz Wo Nicholas Sze
                10.
                		 Add a new DFSAdminCommand for rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
                11.
                		 Support query and finalize rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
                12.
                		 Add new NN startup options for downgrade and rollback using upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
                13.
                		 Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion Sub-task Resolved Brandon Li
                14.
                		 Document new commands and parameters for improved rolling upgrades Sub-task Resolved Tsz Wo Nicholas Sze
                15.
                		 Add a new option for starting standby NN when rolling upgrade is in progress Sub-task Resolved Tsz Wo Nicholas Sze
                16.
                		 Add rolling upgrade infomation to heartbeat response Sub-task Resolved Tsz Wo Nicholas Sze
                17.
                		 Reading edits should not stop at UpgradeMarker for normal restart of the namenode Sub-task Resolved Vinayakumar B
                18.
                		 When rolling upgrade is in progress, NN should only create checkpoint right before the upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
                19.
                		 Should not compare DataNode current layout version with that of NameNode in DataStrorage Sub-task Resolved Brandon Li
                20.
                		 When rolling upgrade is in progress, standby NN should create checkpoint for downgrade. Sub-task Resolved Tsz Wo Nicholas Sze
                21.
                		 Avoid NPE in Datanode heartbeat Sub-task Resolved Vinayakumar B
                22.
                		 Upgrade and rolling upgrade should not be allowed simultaneously Sub-task Resolved Tsz Wo Nicholas Sze
                23.
                		 BlockPoolSliceStorage trash to handle block deletions during rolling upgrade Sub-task Resolved Arpit Agarwal
                24.
                		 Support rollback of rolling upgrade in NameNode and JournalNodes Sub-task Resolved Jing Zhao
                25.
                		 Utilize OOB upgrade message processing for writes Sub-task Resolved Kihwal Lee
                26.
                		 Add rolling upgrade information to fsimage Sub-task Resolved Tsz Wo Nicholas Sze
                27.
                		 Merge Protobuf-based-FSImage code from trunk Sub-task Resolved Jing Zhao
                28.
                		 Fix TestRollingUpgrade Sub-task Resolved Arpit Agarwal
                29.
                		 TestRollingUpgrade#testSecondaryNameNode causes subsequent tests to fail Sub-task Resolved Tsz Wo Nicholas Sze
                30.
                		 Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
                31.
                		 Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
                32.
                		 Fix compilation error after merge Sub-task Resolved Tsz Wo Nicholas Sze
                33.
                		 Create unit tests for downgrade and finalize Sub-task Resolved Haohui Mai
                34.
                		 Rollback does not need to load edits Sub-task Resolved Jing Zhao
                35.
                		 Fix TestEditLog and TestStandbyCheckpoints Sub-task Resolved Jing Zhao
                36.
                		 SimulatedFSDataset#disableAndPurgeTrashStorage should not throw UnsupportedOperationException Sub-task Resolved Jing Zhao
                37.
                		 Fix findbugs warnings in Rolling Upgrade branch Sub-task Resolved Tsz Wo Nicholas Sze
                38.
                		 Fix TestOfflineEditsViewer and NPE in MD5FileUtils Sub-task Resolved Tsz Wo Nicholas Sze
                39.
                		 Fix TestDataNodeRollingUpgrade failure Sub-task Resolved Arpit Agarwal
                40.
                		 Do not create rollback fsimage when it already exists Sub-task Resolved Jing Zhao
                41.
                		 Avoid saving namespace when starting rolling upgrade Sub-task Resolved Jing Zhao
                42.
                		 Add '-rollingUpgrade <rollback|downgrade>' to namenode usage message Sub-task Resolved Vinayakumar B
                43.
                		 Change DFSAdmin for rolling upgrade commands Sub-task Resolved Tsz Wo Nicholas Sze
                44.
                		 Simplify Datanode rollback and downgrade Sub-task Resolved Suresh Srinivas
                45.
                		 add rollingUpgrade information to latest UI Sub-task Resolved Vinayakumar B
                46.
                		 Fix findbug warnings introduced by HDFS-5583 Sub-task Resolved Kihwal Lee
                47.
                		 Fix TestBlockRecovery#testRaceBetweenReplicaRecoveryAndFinalizeBlock Sub-task Resolved Kihwal Lee
                48.
                		 Update datanode replacement policy to make writes more robust Sub-task Resolved Kihwal Lee
                49.
                		 Query the status of rolling upgrade in the preparation stage in TestRollingUpgrade and TestRollingUpgradeRollback Sub-task Resolved Haohui Mai
                50.
                		 Standby NN might not checkpoint when processing the rolling upgrade marker Sub-task Resolved Haohui Mai
                51.
                		 Fix the five findbugs warnings Sub-task Resolved Kihwal Lee
                52.
                		 Test whether the standby NN continues to checkpoint after the prepare stage Sub-task Resolved Haohui Mai
                53.
                		 Test whether the NN will reject the downgrade if it has a fsimage from a newer release Sub-task Resolved Haohui Mai
                54.
                		 Fix TestDFSUpgrade and TestDataNodeRollingUpgrade Sub-task Resolved Jing Zhao
                55.
                		 Secondary NN fails to checkpoint after -rollingUpgrade prepare Sub-task Resolved Jing Zhao
                56.
                		 NN running newer software rejects loading the fsimage during rolling upgrade. Sub-task Resolved Tsz Wo Nicholas Sze
                57.
                		 -rollingUpgrade query hits NPE after the NN restarts Sub-task Resolved Haohui Mai
                58.
                		 DN registration should use DataNodeLayoutVersion instead of NameNodeLayoutVersion Sub-task Resolved Tsz Wo Nicholas Sze
                59.
                		 Allow JournalNode to handle editlog produced by new release with future layoutversion Sub-task Closed Jing Zhao
                60.
                		 Downgrade/Finalize should rename the rollback image instead of purging it Sub-task Resolved Jing Zhao
                61.
                		 Fix rolling upgrade documentation and error messges Sub-task Resolved Tsz Wo Nicholas Sze
                62.
                		 NameNode should not check DataNode layout version Sub-task Closed Brandon Li
                63.
                		 SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type Sub-task Closed Tsz Wo Nicholas Sze

                  Activity

                    People

                    • Assignee:
                      szetszwo Tsz Wo Nicholas Sze
                      Reporter:
                      nroberts Nathan Roberts
                    • Votes:
                      1 Vote for this issue
                      Watchers:
                      36 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: