[HDFS-12943] Consistent Reads from Standby Node

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.3.0, 3.1.4, 3.2.2
• Component/s: hdfs
• Target Version/s: 3.3.0
• Hadoop Flags: Reviewed

Release Note

Observer is a new type of a NameNode in addition to Active and Standby Nodes in HA settings. An Observer Node maintains a replica of the namespace same as a Standby Node. It additionally allows execution of clients read requests.

To ensure read-after-write consistency within a single client, a state ID is introduced in RPC headers. The Observer responds to the client request only after its own state has caught up with the client’s state ID, which it previously received from the Active NameNode.

Clients can explicitly invoke a new client protocol call msync(), which ensures that subsequent reads by this client from an Observer are consistent.

A new client-side ObserverReadProxyProvider is introduced to provide automatic switching between Active and Observer NameNodes for submitting respectively write and read requests.

• Assignee: Konstantin Shvachko
• Reporter: Konstantin Shvachko
• Created: 19/Dec/17 20:16
• Updated: 20/Jan/21 01:56
• Resolved: 01/Nov/19 00:27

Description

StandbyNode in HDFS is a replica of the active NameNode. The states of the NameNodes are coordinated via the journal. It is natural to consider StandbyNode as a read-only replica. As with any replicated distributed system the problem of stale reads should be resolved. Our main goal is to provide reads from standby in a consistent way in order to enable a wide range of existing applications running on top of HDFS.

Issue Links

• breaks: HDFS-14435 ObserverReadProxyProvider is unable to properly fetch HAState from Standby NNs
• incorporates: HDFS-15751 Add documentation for msync() API to filesystem.md
• is related to: HDFS-14245 Class cast error in GetGroups with ObserverReadProxyProvider
• is related to: HDFS-13664 Refactor ConfiguredFailoverProxyProvider to make inheritance easier
• is related to: HDFS-10702 Add a Client API and Proxy Provider to enable stale read from Standby
• is related to: HDFS-6440 Support more than 2 NameNodes
• is related to: HADOOP-17477 [SBN read] Implement msync() for ViewFS
• is related to: HDFS-14205 Backport HDFS-6440 to branch-2
• is related to: HDFS-10519 Add a configuration option to enable in-progress edit log tailing
• is related to: HDFS-13735 Make QJM HTTP URL connection timeout configurable
• is related to: HDFS-13814 Remove super user privilege requirement for NameNode.getServiceStatus
• relates to: HDFS-14272 [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
• relates to: HDFS-14279 [SBN Read] Race condition in ObserverReadProxyProvider
• relates to: HDFS-14347 Restore a comment line mistakenly removed in ProtobufRpcEngine
• relates to: HDFS-14204 Backport HDFS-12943 to branch-2
• relates to: HDFS-14211 [Consistent Observer Reads] Allow for configurable "always msync" mode
• relates to: HDFS-14250 [SBN read] msync should sync with active NameNode to fetch the latest stateID
• relates to: HDFS-14271 [SBN read] StandbyException is logged if Observer is the first NameNode
• relates to: HDFS-14573 Backport Standby Read to branch-3

Sub-Tasks

• Support observer reads for WebHDFS: Open
• Allow Observer to participate in NameNode failover: Open
• NameNodeRpcServer getEditsFromTxid assumes it is run on active NameNode: Open
• Add a configuration to turn on/off observer reads: Open
• [SBN read] reportBadBlock is rejected by Observer.: Open