Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14272

[SBN read] ObserverReadProxyProvider should sync with active txnID on startup

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: tools
    • Labels:
      None
    • Environment:

      CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby + SSL + Kerberos + RPC encryption

      Description

      It is typical for integration tests to create some files and then check their existence. For example, like the following simple bash script:

      # hdfs dfs -touchz /tmp/abc
      # hdfs dfs -ls /tmp/abc
      

      The test executes HDFS bash command sequentially, but it may fail with Consistent Standby Read because the -ls does not find the file.

      Analysis: the second bash command, while launched sequentially after the first one, is not aware of the state id returned from the first bash command. So ObserverNode wouldn't wait for the the edits to get propagated, and thus fails.

      I've got a cluster where the Observer has tens of seconds of RPC latency, and this becomes very annoying. (I am still trying to figure out why this Observer has such a long RPC latency. But that's another story.)

        Attachments

        1. HDFS-14272.002.patch
          8 kB
          Erik Krogen
        2. HDFS-14272.001.patch
          12 kB
          Erik Krogen
        3. HDFS-14272.000.patch
          6 kB
          Erik Krogen

          Issue Links

            Activity

              People

              • Assignee:
                xkrogen Erik Krogen
                Reporter:
                weichiu Wei-Chiu Chuang
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: