[AMBARI-13355] Journal node went in critical state on Ambari : message on UI : Connection failed: [Errno 111] Connection refused to 0.0.0.0:8480

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Assignee: Andrew Onischuk
• Reporter: Andrew Onischuk
• Created: 08/Oct/15 12:09
• Updated: 29/Jul/16 06:39
• Resolved: 12/Oct/15 14:48

Description

Ambari UI is showing alert for HDFS->Journal Node Process.
By alert definition "This host-level alert is triggered if the JournalNode
process cannot be confirmed to be up and listening on the network."

When i check on one of the journal node last successful call was made at -
2015-09-29 08:07:11,892

2015-09-29 08:07:11,892 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(145)) - Edits file /grid/0/hadoop/hdfs/namenode/current/edits_0000000000000002757-0000000000000002781, http://scale-test-20-nodes-part1-1.novalocal:8480/getJournal?jid=nameservice&segmentTxId=2757&storageInfo=-63%3A1242882785%3A0%3ACID-6fd83300-71b7-41e3-aa2f-c40614cae059, http://scale-test-20-nodes-part1-11.novalocal:8480/getJournal?jid=nameservice&segmentTxId=2757&storageInfo=-63%3A1242882785%3A0%3ACID-6fd83300-71b7-41e3-aa2f-c40614cae059 of size 1048576 edits # 25 loaded in 0 seconds

But the JournalNode process was running

[root@scale-test-20-nodes-part1-1 hdfs]# /usr/jdk64/jdk1.7.0_67/bin/jps
21731
23257 DFSZKFailoverController
3220 ApplicationHistoryServer
25424 QuorumPeerMain
27350 NodeManager
23011
5427 – process information unavailable
26990 ResourceManager
5354 Application
27938 Jps
6385 Application
22143 JournalNode
23514 NameNode
3176 HMaster

Issue Links

• relates to: AMBARI-14263 History Server monitoring doesn't take into account HTTPS port
• links to: ReviewBoard