[HDFS-8662] In failover state, HDFS commands fail on "Retrying connect to server:<master1>"

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Not A Problem
• Affects Version/s: 2.3.0
• Component/s: ha

Environment

3-master cluster with 3 data nodes and 1 edge node.

• Assignee: Unassigned
• Reporter: Lakshmi VS
• Created: 24/Jun/15 22:30
• Updated: 18/Oct/16 00:32
• Resolved: 18/Oct/16 00:32

Description

Steps (on a 3M cluster):
1. Poweroff master1.
2. Failover to master2.
3. On HDFS commands, there are messages indicating failed attempt to connect to master1, though namenode is active and running on master2.

infinity2:~ # su - hdfs -c "hdfs haadmin -getServiceState nn2"
active
infinity2:~ # su hdfs -c "hdfs dfs -ls /"
15/06/24 15:28:15 INFO ipc.Client: Retrying connect to server: infinity1.labs.teradata.com/39.0.24.1:8020. Already tried 0 time(s); retry policy is RetryPolicy[MultipleLinearRandomRetry[6x10000ms, 10x60000ms], TryOnceThenFail]
15/06/24 15:28:31 INFO ipc.Client: Retrying connect to server: infinity1.labs.teradata.com/39.0.24.1:8020. Already tried 1 time(s); retry policy is RetryPolicy[MultipleLinearRandomRetry[6x10000ms, 10x60000ms], TryOnceThenFail]
Found 7 items
drwxrwxrwx   - yarn   hadoop          0 2015-06-24 14:42 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2015-06-24 06:16 /apps
drwxr-xr-x   - hdfs   hdfs            0 2015-06-24 06:14 /hdp
drwxr-xr-x   - mapred hdfs            0 2015-06-24 06:15 /mapred
drwxrwxrwx   - mapred hadoop          0 2015-06-24 06:15 /mr-history
drwxrwxrwx   - hdfs   hdfs            0 2015-06-24 06:16 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2015-06-24 06:17 /user
infinity2:~ # ps -fu hdfs
UID        PID  PPID  C STIME TTY          TIME CMD
hdfs     16318     1  0 06:25 ?        00:00:40 /opt/teradata/jvm64/jdk8/bin/java -Dproc_journalnode -Xmx4096m -Dhdp.version=2.3.0.0-2462 -Djava.net.prefe
hdfs     16859     1  0 06:26 ?        00:00:29 /opt/teradata/jvm64/jdk8/bin/java -Dproc_zkfc -Xmx4096m -Dhdp.version=2.3.0.0-2462 -Djava.net.preferIPv4St
hdfs     17791     1  0 06:26 ?        00:02:34 /opt/teradata/jvm64/jdk8/bin/java -Dproc_namenode -Xmx4096m -Dhdp.version=2.3.0.0-2462 -Djava.net.preferIP
infinity2:~ #