[AMBARI-22594] Livy server start fails during EU with 'Address already in use' error

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.1
• Fix Version/s: 2.6.1
• Component/s: ambari-server
• Assignee: Dmitry Lysnichenko
• Reporter: Dmitry Lysnichenko
• Created: 05/Dec/17 14:59
• Updated: 05/Dec/17 18:39
• Resolved: 05/Dec/17 17:34

Description

Observed this issue quite consistently in Ambari-2.6.1 Upgrade ST runs

STR

1. Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 2.6.1.0-129
2. Upgrade Ambari to Target Version: 2.6.1.0-43 | Hash: acbce28fdd119c72625c6beff63fc169de58ba22
3. Regenerate keytabs post Ambari upgrade and this step will restart all services. Here Livy server is operational and gets restarted fine (at timestamp: 09:29)
4. Now register HDP-2.6.4.0-36 version and perform EU. During EU 'Restart Livy server' task happens and reports success (at timestamp: 10:26)
5. However when checking the livy logs - Livy restart reported below exception as the previous process was not killed/stopped

   17/11/21 10:26:22 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@3bc735b3: java.net.BindException: Address already in use
   java.net.BindException: Address already in use
   at sun.nio.ch.Net.bind0(Native Method)
   at sun.nio.ch.Net.bind(Net.java:433)
   at sun.nio.ch.Net.bind(Net.java:425)
   at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
   at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
   at org.apache.livy.server.LivyServer.main(LivyServer.scala)
   Exception in thread "main" java.net.BindException: Address already in use
   at sun.nio.ch.Net.bind0(Native Method)
   at sun.nio.ch.Net.bind(Net.java:433)
   at sun.nio.ch.Net.bind(Net.java:425)
   at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
   

• Post Upgrade, I tried to stop/start Spark as well and Livy still gave same exception; although web Ui reports operation as success (at timestamp: 11:37)
• Finally the web UI shows Livy as down, even though the process is running from the initial step (at timestamp: 09:29)

Issue Links

• links to: review_request_