[HDFS-13234] Remove renew configuration instance in ConfiguredFailoverProxyProvider and reduce memory footprint for client

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs, ha, hdfs-client
• Assignee: Unassigned
• Reporter: Xiaoqiao He
• Created: 06/Mar/18 04:43
• Updated: 12/Mar/18 03:21

Description

The memory footprint of #DFSClient is very considerable in some special scenario since there are many #Configuration instances and occupy much memory resource (In an extreme case, org.apache.hadoop.conf.Configuration occupies over 600MB we meet under HDFS Federation an HA with QJM and there are dozens of NameNodes). I think some new Configuration instance is not necessary. Such as #ConfiguredFailoverProxyProvider initialization.

  public ConfiguredFailoverProxyProvider(Configuration conf, URI uri,
      Class<T> xface, HAProxyFactory<T> factory) {
    this.xface = xface;
    this.conf = new Configuration(conf);
    ......
  }