[LIVY-197] Livy doesn't support RM HA

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: 0.3
• Component/s: Server
• Labels: resource-management
• Environment: Hadoop 2.7.2, Hue 3.10.0, Spark 1.6.1, Livy 0.3 (github snapshot), Cluster secured with Kerberos
• Assignee: Unassigned
• Reporter: Pierre Beauvois
• Created: 22/Jul/16 07:31
• Updated: 13/Jul/17 16:38
• Resolved: 27/Jul/16 16:17

Description

Hi,
I'm currently trying to configure Livy.
I already set up impersonation in the core-site.xml like the following:

<property>
        <name>hadoop.proxyuser.hue.groups</name>
        <value>*</value>
        <description>Allow the user to impersonate any members of group1 and group2</description>
</property>

<property>
        <name>hadoop.proxyuser.hue.hosts</name>
        <value>hue01.bigdata.fr</value>
        <description>Allow the user to connect only from host1 and host2 to impersonate a user</description>
</property>

<property>
        <name>hadoop.proxyuser.livy.groups</name>
        <value>*</value>
        <description>Allow the user to impersonate any members of group1 and group2</description>
</property>
<property>
        <name>hadoop.proxyuser.livy.hosts</name>
        <value>hue01.bigdata.fr</value>
        <description>Allow the user to connect only from host1 and host2 to impersonate a user</description>
</property>

The cluster is also secured with Kerberos so I set the auth type, the principal and the keytab path in livy.conf (attached).

Unfortunately, my livy sessions are not working properly. Sessions are killed after 10 minutes in a "startup" state. My resourcemanager HA is not working for testing purpose and I think it's the source of the error here. The machine "resourcemanager01.bigdata.fr is stopped" but "resourcemanager02.bigdata.fr" is running.
You can find below the stacktrace.

Jul 22 09:02:38 hue01.bigdata.fr livy INFO - com.cloudera.livy.server.LivyServerUsing spark-submit version 1.6.1
Jul 22 09:02:38 hue01.bigdata.fr livy INFO - org.eclipse.jetty.util.logLogging initialized @1536ms
Jul 22 09:02:38 hue01.bigdata.fr livy INFO - com.cloudera.livy.server.LivyServerSPNEGO auth enabled (principal = HTTP/[e-mail])
Jul 22 09:02:38 hue01.bigdata.fr livy INFO - org.eclipse.jetty.server.Serverjetty-9.2.16.v20160414
Jul 22 09:02:39 hue01.bigdata.fr livy INFO - org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandlerLogin using keytab /opt/application/Livy/current/spnego.keytab, for principal HTTP/[e-mail]
Jul 22 09:02:39 hue01.bigdata.fr livy INFO - org.eclipse.jetty.server.handler.ContextHandlerStarted o.e.j.s.ServletContextHandler@6c814dbd{/,file:/src/main/com/cloudera/livy/server,AVAILABLE}
Jul 22 09:02:39 hue01.bigdata.fr livy WARN - org.eclipse.jetty.server.handler.RequestLogHandler!RequestLog
Jul 22 09:02:39 hue01.bigdata.fr livy INFO - org.eclipse.jetty.server.ServerConnectorStarted ServerConnector@57c3eb95{HTTP/1.1}{0.0.0.0:8998}
Jul 22 09:02:39 hue01.bigdata.fr livy INFO - org.eclipse.jetty.server.ServerStarted @2571ms
Jul 22 09:02:39 hue01.bigdata.fr livy INFO - com.cloudera.livy.server.WebServerStarting server on http://hue01.bigdata.fr:8998
Jul 22 09:08:24 hue01.bigdata.fr livy INFO - com.cloudera.livy.server.interactive.InteractiveSessionCreating LivyClient for sessionId: 0
Jul 22 09:08:24 hue01.bigdata.fr livy WARN - com.cloudera.livy.rsc.RSCConfYour hostname, hue01.bigdata.fr, resolves to a loopback address, but we couldn't find any external IP address!
Jul 22 09:08:24 hue01.bigdata.fr livy WARN - com.cloudera.livy.rsc.RSCConfSet livy.rsc.rpc.server.address if you need to bind to another address.
Jul 22 09:08:24 hue01.bigdata.fr livy INFO - com.cloudera.livy.sessions.SessionManagerRegistering new session 0
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher16/07/22 09:15:49 ERROR SparkContext: Error initializing SparkContext.
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncherjava.net.ConnectException: Call From hue01.bigdata.fr/192.168.200.208 to resourcemanager01.bigdata.fr:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.ipc.Client.call(Client.java:1479)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.ipc.Client.call(Client.java:1412)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at com.sun.proxy.$Proxy10.getNewApplication(Unknown Source)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:221)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at java.lang.reflect.Method.invoke(Method.java:606)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at com.sun.proxy.$Proxy11.getNewApplication(Unknown Source)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:219)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:227)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:132)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at py4j.Gateway.invoke(Gateway.java:214)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at py4j.GatewayConnection.run(GatewayConnection.java:209)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at java.lang.Thread.run(Thread.java:745)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncherCaused by: java.net.ConnectException: Connection refused
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011at org.apache.hadoop.ipc.Client.call(Client.java:1451)
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher#011... 28 more
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.ContextLauncher16/07/22 09:15:49 ERROR PythonInterpreter: Process has died with 1
Jul 22 09:15:49 hue01.bigdata.fr livy INFO - com.cloudera.livy.rsc.RSCClientReceived result for b256b2e5-0739-43ab-827a-0d2e9d4f6020