[HDFS-1623] High Availability Framework for HDFS NN

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0-alpha
• Target Version/s: 0.23.3
• Hadoop Flags: Reviewed
• Assignee: Unassigned
• Reporter: Sanjay Radia
• Created: 11/Feb/11 02:45
• Updated: 28/Sep/15 20:58
• Resolved: 03/Mar/12 01:47

Issue Links

• breaks: MAPREDUCE-4057 Compilation error in RAID
• is blocked by: HADOOP-7454 Common side of High Availability Framework (HDFS-1623)
• is related to: HDFS-1149 Lease reassignment is not persisted to edit log
• is related to: HDFS-1064 NN Availability - umbrella Jira
• is related to: HDFS-2064 Warm HA NameNode going Hot
• relates to: HDFS-8693 refreshNamenodes does not support adding a new standby to a running DN
• relates to: HDFS-3399 BookKeeper option support for NN HA
• relates to: HIVE-3056 Create a new metastore tool to bulk update location field in Db/Table/Partition records

Sub-Tasks

• HA: Send block report from datanode to both active and standby namenodes: Resolved
• HA: Datanode fencing mechanism: Resolved
• HA: HDFS clients must handle namenode failover and switch over to the new active namenode.: Resolved
• HA: Introduce active and standby states to the namenode: Resolved
• HA: Support for sharing the namenode state from active to standby.: Resolved
• Client should not timeout during a NN start - esp useful for HA NN: Resolved
• Remove NameNode roles Active and Standby (they become states): Closed
• Merge NameNode roles into NodeType.: Resolved
• Configuration changes for HA namenode: Resolved
• Log newly allocated blocks: Closed
• HA: Checkpointing in an HA setup: Resolved
• Start/stop appropriate namenode internal services during transition to active and standby: Resolved
• Generalize the HAServiceProtocol interface: Resolved
• HA: Enable the configuration of multiple HA cluster addresses: Resolved
• Mark appropriate methods of ClientProtocol with the idempotent annotation: Resolved
• Add tests for Namenode active standby states: Resolved
• getServerDefaults and getStats don't check operation category: Resolved
• Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231: Resolved
• Add HA-related metrics: Resolved
• NameNode needs to add HAServiceProtocol to its RPC Server: Resolved
• HA: NN fails to start since it tries to start secret manager in safemode: Resolved
• Scope dfs.ha.namenodes config by nameservice: Resolved
• Add protobuf service and implementation for HAServiceProtocol: Resolved
• HA: MiniDFSCluster support to mix and match federation with HA: Resolved
• HA: Balancer support for HA namenodes: Resolved
• NN should log newly-allocated blocks without losing BlockInfo: Resolved
• HA: don't initialize replication queues until entering Active mode: Resolved
• HA: handle refreshNameNodes in federated HA clusters: Resolved
• Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse: Resolved
• HA: fix TestDFSUpgrade on HA branch: Resolved
• HA: Add test case for hot standby capability: Resolved
• HA: ConfiguredFailoverProxyProvider doesn't correctly stop ProtocolTranslators: Resolved
• HA: TestDfsOverAvroRpc failing after introduction of HeartbeatResponse type: Resolved
• HA: BPOfferService.verifyAndSetNamespaceInfo needs to be synchronized: Resolved
• HA: determine DN's view of which NN is active based on heartbeat responses: Resolved
• Standby needs to ingest latest edit logs before transitioning to active`: Resolved
• HA: Fix NN Active->Standby transition: Resolved
• HA: NN should throw StandbyException in response to RPCs in STANDBY state: Resolved
• HA: Web UI should indicate the NN state: Resolved
• HA: When a FailoverProxyProvider is used, DFSClient should not retry connection ten times before failing over: Resolved
• Add interface to query current state to HAServiceProtocol: Resolved
• DFSClient should construct failover proxy with exponential backoff: Resolved
• HA: When a FailoverProxyProvider is used, Client should not retry for 45 times(hard coded value) if it is timing out to connect to server.: Resolved
• Authority-based lookup of proxy provider fails if path becomes canonicalized: Resolved
• Fix up some failing unit tests on HA branch: Resolved
• HA: write tests for quota tracking and HA: Resolved
• HA: BookKeeperEditLogInputStream doesn't implement isInProgress(): Resolved
• HA: Tests and fixes for pipeline targets and replica recovery: Resolved
• HA: Bugs related to failover from/into safe-mode: Resolved
• Synchronization issues around state transition: Resolved
• HA: Appropriately handle error conditions in EditLogTailer: Resolved
• HA : An alternative approach to clients handling Namenode failover.: Resolved
• HA: Fix test cases which use standalone FSNamesystems: Resolved
• HA: Configuration needs to allow different dfs.http.addresses for each HA NN: Resolved
• HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 nameSpaceDirs to NN2 nameSpaceDirs: Resolved
• NN web UI can throw NPE after startup, before standby state is entered: Resolved
• HA: Refactor shared HA-related test code into HATestUtils class: Resolved
• Add support for the standby in the bin scripts: Resolved
• Document HA configuration and CLI: Resolved
• HA: add tests for multiple shared edits dirs: Resolved
• HA: support 2NN with SBN: Resolved
• HA: Automatically trigger log rolls periodically on the active NN: Resolved
• FSEditLog.selectinputStreams is reading through in-progress streams even when non-in-progress are requested: Resolved
• HA: observed dataloss in replication stress test: Resolved
• HA: entering safe mode after starting SBN can NPE: Resolved
• HA: exit if multiple shared dirs are configured: Resolved
• Warm standby does not read the in_progress edit log: Resolved
• TestCheckpoint is timing out: Resolved
• Standby namenode stuck in safemode during a failover: Resolved
• HA: test for case where standby partially reads log and then performs checkpoint: Resolved
• HA: ConfiguredFailoverProxyProvider should support NameNodeProtocol: Resolved
• HA: When HA is enabled with a shared edits dir, that dir should be marked required: Resolved
• HA: On transition to active, standby should not swallow ELIE: Resolved
• HA: reading edit logs from an earlier version leaves blocks in under-construction state: Resolved
• HA: TestStandbyCheckpoints.testBothNodesInStandbyState fails intermittently: Resolved
• HA: Add lease recovery handling to HA: Resolved
• TestHAAdmin.testFailover is failing: Resolved
• HA: Make fsck work: Resolved
• HA: Active NN may purge edit log files before standby NN has a chance to read them: Resolved
• HA: Standby NN takes a long time to recover from a dead DN starting up: Resolved
• TestDFSClientFailover shuold use simpleHATopology.: Resolved
• SBN should not mark blocks under-replicated when exiting safemode: Resolved
• HA: Add a test for a federated cluster with HA NNs: Resolved
• ConfiguredFailoverProxyProvider should be moved to common.: Resolved
• Service level authorizartion for HAServiceProtocol: Resolved
• HA: haadmin should use namenode ids: Resolved
• Add test to verify that delegation tokens are honored after failover.: Resolved
• HA: Client should fail if a failover occurs which switches block pool ID: Resolved
• When becoming active, NN should treat all leases as freshly renewed: Resolved
• Document new HA-related configs in hdfs-default.xml: Resolved
• Add a simple sanity check for HA config: Resolved
• HA: Transition to active can cause NN deadlock: Resolved
• HA: failover does not succeed if prior NN died just after creating an edit log segment: Resolved
• HA: Fix ConfiguredFailoverProxyProvider for federation: Resolved
• HA: HAAdmin does not work if security is enabled: Resolved
• HA: TestSafeMode#testNoExtensionIfNoBlocks is failing: Resolved
• SBN should not allow browsing of the file system via web UI: Resolved
• HA: NN fails to start if the shared edits dir is marked required: Resolved
• DFSUtil.getSuffixIDs silently ignores exception in NetUtils.createSocketAddr: Resolved
• LOCAL_ADDRESS_MATCHER.match has NPE when called from DFSUtil.getSuffixIDs when the host is incorrect: Resolved
• HA: TestDFSRollback#testRollback is failing: Resolved
• HA: checkpointing should verify that the dfs.http.address has been configured to a non-loopback for peer NN: Resolved
• Failures observed if dfs.edits.dir and shared.edits.dir have same directories.: Resolved
• Standby does not start up due to a gap in transaction id: Resolved
• Standby namenode gets a "cannot lock storage" exception during startup: Resolved
• HA: Remove some INFO level logging accidentally left around: Resolved
• HA: edit log should log to shared dirs before local dirs: Resolved
• HA: DFSUtil#getSuffixIDs should skip unset configurations: Resolved
• HA: automatically determine the nameservice Id if only one nameservice is configured: Resolved
• Starting delegation token manager during safemode fails: Resolved
• HA: Improvements for SBN web UI - not show under-replicated/missing blocks: Resolved
• HA: Client support for getting delegation tokens to an HA cluster: Resolved
• HA: Standby NN NPE when shared edits dir is deleted: Resolved
• HA: NPE if shared edits directory is not available during failover: Resolved
• HA: Inaccessible shared edits dir not getting removed from FSImage storage dirs upon error: Resolved
• HA: Namenode not shutting down when shared edits dir is inaccessible: Resolved
• HA: TestFailureOfSharedDir.testFailureOfSharedDir() has race condition: Resolved
• HA: haadmin should not work if run by regular user: Resolved
• HA: fix remaining TODO items: Resolved
• HA: close out operation categories: Resolved
• HA: Standby checkpointing fails to authenticate in secure cluster: Resolved
• HA: enable hadoop security authorization for haadmin / protocols: Resolved
• HA: ConfiguredFailoverProxyProvider should not create a NameNode proxy with an underlying retry proxy: Resolved
• HA: stress test and fixes for block synchronization: Resolved
• HA: Allow configs to be scoped to all NNs in the nameservice: Resolved
• HA: Shared edits dir property should be suffixed with nameservice and namenodeID: Resolved
• HA: TestDFSHAAdmin needs tests with MiniDFSCluster: Resolved
• HA: TestHAStateTransitions fails on Windows: Resolved
• HA: NullPointerException while formatting NameNode(After Configuring HA): Resolved
• HA: TestActiveStandbyElectorRealZK fails if build dir does not exist: Resolved
• HA: On startup NN throws an NPE in the metrics system: Resolved
• HA: NN throws NPE during shutdown if it fails to startup: Resolved
• HA: NN should not start with upgrade option or with a pending an unfinalized upgrade: Resolved
• HA: IllegalStateException during standby startup in getCurSegmentTxId: Resolved
• HA: Sweep for remaining proxy construction which doesn't go through failover path: Resolved
• HA: TestDFSUtil is failing: Resolved
• HA: small optimization building incremental block report: Resolved
• HA: re-enable NO_ACK optimization for block deletion: Resolved
• HA: MiniDFSCluster does not delete standby NN name dirs during format: Resolved
• HA: TestBalancerWithHANameNodes is failing: Resolved
• HA: Balancer should use logical uri for creating failover proxy with HA enabled.: Resolved
• HA: BackupNode#checkOperation should permit CHECKPOINT operations: Resolved
• Get performance on HA branch to match trunk: Resolved
• HA: NameNode format doesn't pick up dfs.namenode.name.dir.NameServiceId configure: Resolved
• HA: Implement a simple NN health check: Resolved
• HA: fix failure of TestFileAppendRestart due to OP_UPDATE_BLOCKS: Resolved
• HA: Address findbugs and javadoc warnings on branch: Resolved
• HA: NPE in FSNamesystem when in safe mode: Resolved