[AMBARI-17878] after upgrade is finalized add service operation hang in Preparing to execute Set Keytab

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: ambari-server
• Assignee: Dmitry Lysnichenko
• Reporter: Dmitry Lysnichenko
• Created: 25/Jul/16 10:01
• Updated: 25/Jul/16 18:47
• Resolved: 25/Jul/16 16:15

Description

add service operation hangs in Preparing to execute Set Keytab

1. install ambari2.4.0 with stack 2.4
2. express upgrade to stack 2.5
3. finalize the upgrade
4. add new services ( kms,atlas,slider)

Null pointer exception is thrown in ambari-server logs

18 Jul 2016 07:12:32,025 ERROR [ambari-metrics-retrieval-service-thread-105] MetricsRetrievalService:421 - Unable to retrieve metrics from http://host:8744/api/v1/cluster/summary. Subsequent failures will be suppressed from the log for 5 minutes.
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218)
at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:142)
at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:116)
at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:121)
at org.apache.ambari.server.state.services.MetricsRetrievalService$MetricRunnable.run(MetricsRetrievalService.java:382)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

18 Jul 2016 07:42:39,979  WARN [ambari-action-scheduler] ActionScheduler:202 - Exception received
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
at org.apache.ambari.server.agent.ActionQueue.getQueue(ActionQueue.java:52)
at org.apache.ambari.server.agent.ActionQueue.enqueue(ActionQueue.java:61)
at org.apache.ambari.server.actionmanager.ActionScheduler.cancelCommandOnTimeout(ActionScheduler.java:1084)
at org.apache.ambari.server.actionmanager.ActionScheduler.processInProgressStage(ActionScheduler.java:725)
at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:290)
at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:197)
at java.lang.Thread.run(Thread.java:745)

Note: this is an upgraded cluster from hdp-2.4 to hdp-2.5 , addition of the service is done after upgrade.

Issue Links

• links to: review_request_