[HDFS-5566] HA namenode with QJM created from org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider should implement Closeable

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid

Environment

hadoop-2.2.0
hbase-0.96

• Assignee: Jimmy Xiang
• Reporter: Henry Hung
• Created: 26/Nov/13 03:12
• Updated: 16/Dec/13 23:49
• Resolved: 16/Dec/13 23:49

Description

When using hbase-0.96 with hadoop-2.2.0, stopping master/regionserver node will result in Cannot close proxy - is not Closeable or does not provide closeable invocation.

Mail Archive

My hadoop-2.2.0 configured as HA namenode with QJM, the configuration is like this:

  <property>
    <name>dfs.nameservices</name>
    <value>hadoopdev</value>
  </property>
  <property>
    <name>dfs.ha.namenodes.hadoopdev</name>
    <value>nn1,nn2</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.hadoopdev.nn1</name>
    <value>fphd9.ctpilot1.com:9000</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.hadoopdev.nn1</name>
    <value>fphd9.ctpilot1.com:50070</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.hadoopdev.nn2</name>
    <value>fphd10.ctpilot1.com:9000</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.hadoopdev.nn2</name>
    <value>fphd10.ctpilot1.com:50070</value>
  </property>
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://fphd8.ctpilot1.com:8485;fphd9.ctpilot1.com:8485;fphd10.ctpilot1.com:8485/hadoopdev</value>
  </property>
  <property>
    <name>dfs.client.failover.proxy.provider.hadoopdev</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>shell(/bin/true)</value>
  </property>
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/data/hadoop/hadoop-data-2/journal</value>
  </property>
  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>fphd1.ctpilot1.com:2222</value>
  </property>

I traced the code and found out that when stopping the hbase master node, it will try invoke method "close" on namenode, but the instance that created from org.apache.hadoop.hdfs.NameNodeProxies.createProxy with failoverProxyProviderClass org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider do not have the Closeable interface.

If we use the Non-HA case, the created instance will be org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB that implement Closeable.

TL;DR;
With hbase connecting to hadoop HA namenode, when stopping the hbase master or regionserver, it couldn't find the close method to gracefully close namenode session.

Issue Links

• duplicates: HBASE-10029 Proxy created by HFileSystem#createReorderingProxy() should properly close when connecting to HA namenode
• is depended upon by: HBASE-10159 Replaced deprecated interface Closeable