[HDFS-5535] Umbrella jira for improved HDFS rolling upgrades

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 3.0.0-alpha1
• Fix Version/s: 2.4.0
• Component/s: datanode, ha, hdfs-client, namenode
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Nathan Roberts
• Created: 20/Nov/13 19:20
• Updated: 12/May/16 18:15
• Resolved: 11/Mar/14 00:20

Description

In order to roll a new HDFS release through a large cluster quickly and safely, a few enhancements are needed in HDFS. An initial High level design document will be attached to this jira, and sub-jiras will itemize the individual tasks.

Issue Links

• is related to: HDFS-7230 Add rolling downgrade documentation
• relates to: HDFS-6800 Support Datanode layout changes with rolling upgrade

Sub-Tasks

• Revist upgrade snapshots, roll back, finalize to enable rolling upgrades: Resolved
• Make replication queue initialization asynchronous: Resolved
• Improve datanode startup time: Resolved
• Consider supporting a mechanism to allow datanodes to drain outstanding work during rolling upgrade: Resolved
• Make DN send an OOB Ack on shutdown before restarting: Resolved
• Add OOB upgrade response and client-side logic for reads: Resolved
• Provide admin commands for data node upgrade: Resolved
• Add quick-restart option for datanode: Resolved
• Support upgrade marker in editlog streams: Resolved
• Add a new DFSAdminCommand for rolling upgrade: Resolved
• Support query and finalize rolling upgrade: Resolved
• Add new NN startup options for downgrade and rollback using upgrade marker: Resolved
• Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion: Resolved
• Document new commands and parameters for improved rolling upgrades: Resolved
• Add a new option for starting standby NN when rolling upgrade is in progress: Resolved
• Add rolling upgrade infomation to heartbeat response: Resolved
• Reading edits should not stop at UpgradeMarker for normal restart of the namenode: Resolved
• When rolling upgrade is in progress, NN should only create checkpoint right before the upgrade marker: Resolved
• Should not compare DataNode current layout version with that of NameNode in DataStrorage: Resolved
• When rolling upgrade is in progress, standby NN should create checkpoint for downgrade.: Resolved
• Avoid NPE in Datanode heartbeat: Resolved
• Upgrade and rolling upgrade should not be allowed simultaneously: Resolved
• BlockPoolSliceStorage trash to handle block deletions during rolling upgrade: Resolved
• Support rollback of rolling upgrade in NameNode and JournalNodes: Resolved
• Utilize OOB upgrade message processing for writes: Resolved
• Add rolling upgrade information to fsimage: Resolved
• Merge Protobuf-based-FSImage code from trunk: Resolved
• Fix TestRollingUpgrade: Resolved
• TestRollingUpgrade#testSecondaryNameNode causes subsequent tests to fail: Resolved
• Fix rollback of rolling upgrade in NameNode HA setup: Resolved
• Fix rollback of rolling upgrade in NameNode HA setup: Resolved
• Fix compilation error after merge: Resolved
• Create unit tests for downgrade and finalize: Resolved
• Rollback does not need to load edits: Resolved
• Fix TestEditLog and TestStandbyCheckpoints: Resolved
• SimulatedFSDataset#disableAndPurgeTrashStorage should not throw UnsupportedOperationException: Resolved
• Fix findbugs warnings in Rolling Upgrade branch: Resolved
• Fix TestOfflineEditsViewer and NPE in MD5FileUtils: Resolved
• Fix TestDataNodeRollingUpgrade failure: Resolved
• Do not create rollback fsimage when it already exists: Resolved
• Avoid saving namespace when starting rolling upgrade: Resolved
• Add '-rollingUpgrade <rollback|downgrade>' to namenode usage message: Resolved
• Change DFSAdmin for rolling upgrade commands: Resolved
• Simplify Datanode rollback and downgrade: Resolved
• add rollingUpgrade information to latest UI: Resolved
• Fix findbug warnings introduced by HDFS-5583: Resolved
• Fix TestBlockRecovery#testRaceBetweenReplicaRecoveryAndFinalizeBlock: Resolved
• Update datanode replacement policy to make writes more robust: Resolved
• Query the status of rolling upgrade in the preparation stage in TestRollingUpgrade and TestRollingUpgradeRollback: Resolved
• Standby NN might not checkpoint when processing the rolling upgrade marker: Resolved
• Fix the five findbugs warnings: Resolved
• Test whether the standby NN continues to checkpoint after the prepare stage: Resolved
• Test whether the NN will reject the downgrade if it has a fsimage from a newer release: Resolved
• Fix TestDFSUpgrade and TestDataNodeRollingUpgrade: Resolved
• Secondary NN fails to checkpoint after -rollingUpgrade prepare: Resolved
• NN running newer software rejects loading the fsimage during rolling upgrade.: Resolved
• -rollingUpgrade query hits NPE after the NN restarts: Resolved
• DN registration should use DataNodeLayoutVersion instead of NameNodeLayoutVersion: Resolved
• Allow JournalNode to handle editlog produced by new release with future layoutversion: Closed
• Downgrade/Finalize should rename the rollback image instead of purging it: Resolved
• Fix rolling upgrade documentation and error messges: Resolved
• NameNode should not check DataNode layout version: Closed
• SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type: Closed