Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12943

Consistent Reads from Standby Node

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.3.0, 3.1.4, 3.2.2
    • Component/s: hdfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Observer is a new type of a NameNode in addition to Active and Standby Nodes in HA settings. An Observer Node maintains a replica of the namespace same as a Standby Node. It additionally allows execution of clients read requests.

      To ensure read-after-write consistency within a single client, a state ID is introduced in RPC headers. The Observer responds to the client request only after its own state has caught up with the client’s state ID, which it previously received from the Active NameNode.

      Clients can explicitly invoke a new client protocol call msync(), which ensures that subsequent reads by this client from an Observer are consistent.

      A new client-side ObserverReadProxyProvider is introduced to provide automatic switching between Active and Observer NameNodes for submitting respectively write and read requests.
      Show
      Observer is a new type of a NameNode in addition to Active and Standby Nodes in HA settings. An Observer Node maintains a replica of the namespace same as a Standby Node. It additionally allows execution of clients read requests. To ensure read-after-write consistency within a single client, a state ID is introduced in RPC headers. The Observer responds to the client request only after its own state has caught up with the client’s state ID, which it previously received from the Active NameNode. Clients can explicitly invoke a new client protocol call msync(), which ensures that subsequent reads by this client from an Observer are consistent. A new client-side ObserverReadProxyProvider is introduced to provide automatic switching between Active and Observer NameNodes for submitting respectively write and read requests.

      Description

      StandbyNode in HDFS is a replica of the active NameNode. The states of the NameNodes are coordinated via the journal. It is natural to consider StandbyNode as a read-only replica. As with any replicated distributed system the problem of stale reads should be resolved. Our main goal is to provide reads from standby in a consistent way in order to enable a wide range of existing applications running on top of HDFS.

        Attachments

        1. ConsistentReadsFromStandbyNode.pdf
          394 kB
          Konstantin Shvachko
        2. ConsistentReadsFromStandbyNode.pdf
          396 kB
          Konstantin Shvachko
        3. TestPlan-ConsistentReadsFromStandbyNode.pdf
          79 kB
          Konstantin Shvachko
        4. HDFS-12943-001.patch
          328 kB
          Konstantin Shvachko
        5. HDFS-12943-002.patch
          354 kB
          Konstantin Shvachko
        6. HDFS-12943-003.patch
          353 kB
          Konstantin Shvachko
        7. HDFS-12943-004.patch
          353 kB
          Konstantin Shvachko

          Issue Links

            Activity

              People

              • Assignee:
                shv Konstantin Shvachko
                Reporter:
                shv Konstantin Shvachko
              • Votes:
                4 Vote for this issue
                Watchers:
                76 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: