Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12943 Consistent Reads from Standby Node
  3. HDFS-13923

Add a configuration to turn on/off observer reads

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In some situations having a config to turn on/off observer reads dynamically may be useful. For instance, some applications may be sensitive for data freshness and want to always reach directly to active NN. In a more complex scenario, services such as Presto may want to apply observer reads for different types of queries. In this case, simply change dfs.client.failover.proxy.provider.<nameservice> may not be enough, since with FileSystem cache (which is usually turned on) will ignore the change and still use the same FileSystem object.

      Here I'm proposing to add a flag in HdfsClientConfigKeys, such as dfs.client.observer.reads.enabled, that can be used to dynamically turn on/off observer reads. The FileSystem cache key should also take account of this flag in its hashCode and equals impl, so that different FileSystem objects will be used depending on the flag.

       

      cc Konstantin Shvachko, Erik KrogenChen Liang, Plamen Jeliazkov for discussion.

        Attachments

          Activity

            People

            • Assignee:
              shwetayakkali Shweta
              Reporter:
              csun Chao Sun
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated: