[LIVY-700] Connection refused from YARN when impersonation enabled

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.6.0
• Fix Version/s: 0.9.0
• Component/s: Interpreter
• Labels: YARN impersonation
• Environment: Cloudera 5.7 Hadoop 2.6 Spark 2.4.x Centos7 Livy 0.6 Zeppelin 0.8.2
• Flags: Important
• Assignee: Unassigned
• Reporter: Nicola Marangoni
• Created: 24/Oct/19 13:17
• Updated: 12/Nov/22 22:09

Description

I have Livy and Zeppelin running under the user "zeppelin" on an edge node.

The Cloudera-cluster is not kerberized. Livy is set in yarn client mode.

When the impersonation is not enabled the Livy interpreter works, and a YARN job ist started on the cluster using the user "zeppelin".

When livy.impersonation.enabled = true Livy cannot connect to the YARN RM:

19/10/24 14:40:58 INFO utils.LineBufferedStream: 19/10/24 14:40:58 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm271
19/10/24 14:40:58 INFO utils.LineBufferedStream: 19/10/24 14:40:58 INFO retry.RetryInvocationHandler: Exception while invoking getClusterMetrics of class ApplicationClientProtocolPBClientImpl over rm271 after 1 fail over attempts. Trying to fail over immediately.
19/10/24 14:40:58 INFO utils.LineBufferedStream: 19/10/24 14:40:58 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm270
19/10/24 14:40:58 INFO utils.LineBufferedStream: 19/10/24 14:40:58 INFO retry.RetryInvocationHandler: Exception while invoking getClusterMetrics of class ApplicationClientProtocolPBClientImpl over rm270 after 2 fail over attempts. Trying to fail over after sleeping for 374ms.
19/10/24 14:40:58 INFO utils.LineBufferedStream: java.net.ConnectException: Call From <localhost>/123.123.123.123 to yarn.rm.host.name:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

This error happen after the spark driver is successfully started.

Spark simply doesn't get any resources from YARN.

Zeppelin is configured to use FreeIPA over LDAP for login. The user I use to login to the zeppelin UI has even more rights on the cluster and if I start a spark-shell on the same cluster with that user I get a YARN job without any issue.