Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-700

Connection refused from YARN when impersonation enabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0
    • Fix Version/s: None
    • Component/s: Interpreter
    • Environment:
      Cloudera 5.7 Hadoop 2.6 Spark 2.4.x Centos7 Livy 0.6 Zeppelin 0.8.2
    • Flags:
      Important

      Description

      I have Livy and Zeppelin running under the user "zeppelin" on an edge node.

      The Cloudera-cluster is not kerberized. Livy is set in yarn client mode.

      When the impersonation is not enabled the Livy interpreter works, and a YARN job ist started on the cluster using the user "zeppelin".

      When livy.impersonation.enabled = true Livy cannot connect to the YARN RM:

      19/10/24 14:40:58 INFO utils.LineBufferedStream: 19/10/24 14:40:58 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm271
      19/10/24 14:40:58 INFO utils.LineBufferedStream: 19/10/24 14:40:58 INFO retry.RetryInvocationHandler: Exception while invoking getClusterMetrics of class ApplicationClientProtocolPBClientImpl over rm271 after 1 fail over attempts. Trying to fail over immediately.
      19/10/24 14:40:58 INFO utils.LineBufferedStream: 19/10/24 14:40:58 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm270
      19/10/24 14:40:58 INFO utils.LineBufferedStream: 19/10/24 14:40:58 INFO retry.RetryInvocationHandler: Exception while invoking getClusterMetrics of class ApplicationClientProtocolPBClientImpl over rm270 after 2 fail over attempts. Trying to fail over after sleeping for 374ms.
      19/10/24 14:40:58 INFO utils.LineBufferedStream: java.net.ConnectException: Call From <localhost>/123.123.123.123 to yarn.rm.host.name:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

      This error happen after the spark driver is successfully started.

      Spark simply doesn't get any resources from YARN.

      Zeppelin is configured to use FreeIPA over LDAP for login. The user I use to login to the zeppelin UI has even more rights on the cluster and if I start a spark-shell on the same cluster with that user I get a YARN job without any issue.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              nicolamarangoni Nicola Marangoni
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: