Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-372

Livy Doesn't work with Spark on Mesos in cluster mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.3
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None
    • Environment:
      Spark-2.1.0, Mesos-1.2.0

      Description

      Client mode - it works.
      But it doesnt work with cluster mode:

      Added to livy.conf:
      livy.spark.master=mesos://node1:5050
      livy.spark.deployMode=cluster

      Added to hue.ini:
      [spark]
      livy_server_host=node1
      livy_server_port=8998

      Livy logs:
      17/06/16 21:00:26 INFO StateStore$: Using BlackholeStateStore for recovery.
      17/06/16 21:00:26 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
      17/06/16 21:00:26 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
      17/06/16 21:00:26 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
      17/06/16 21:00:26 INFO WebServer: Starting server on http://node1:8998
      17/06/16 22:04:04 INFO LivyServer: Shutting down Livy server.
      17/06/16 22:04:07 INFO StateStore$: Using BlackholeStateStore for recovery.
      17/06/16 22:04:07 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
      17/06/16 22:04:07 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
      17/06/16 22:04:07 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
      17/06/16 22:04:07 INFO WebServer: Starting server on http://node1:8998
      17/06/16 22:05:07 INFO InteractiveSession$: Creating LivyClient for sessionId: 0
      17/06/16 22:05:07 WARN RSCConf: Your hostname, node1, resolves to a loopback address, but we couldn't find any external IP addre$
      17/06/16 22:05:07 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address. (edited)

      [5:01]
      7/06/16 22:05:10 WARN ContextLauncher: Child process exited with code 1.
      17/06/16 22:05:10 ERROR RSCClient: Failed to connect to context.
      java.io.IOException: Child process exited with code 1.
      at com.cloudera.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:416)
      at com.cloudera.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:490)
      at java.lang.Thread.run(Thread.java:748)
      17/06/16 22:05:10 INFO RSCClient: Failing pending job 8a53d717-773c-434e-807f-3ceb46138a66 due to shutdown.
      17/06/16 22:05:10 INFO InteractiveSession: Stopping InteractiveSession 0...
      17/06/16 22:05:10 INFO InteractiveSession: Stopped InteractiveSession 0.
      17/06/16 22:05:10 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.io.IOException: Ch$
      17/06/16 22:05:12 INFO InteractiveSession: Stopping InteractiveSession 0...
      17/06/16 22:05:12 INFO InteractiveSession: Stopped InteractiveSession 0.
      17/06/16 22:12:45 WARN HttpParser: Illegal character 0x16 in state=START for buffer HeapByteBuffer@5bf799a9[p=1,l=171,c=16384,r=$
      17/06/16 22:12:45 WARN HttpParser: badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@10e07e9e{r=0,c=false,a=IDLE,ur$
      17/06/16 22:58:15 WARN HttpParser: Illegal character 0x16 in state=START for buffer HeapByteBuffer@5bf799a9[p=1,l=171,c=16384,r=$
      17/06/16 22:58:15 WARN HttpParser: badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@2ef0a110{r=0,c=false,a=ID

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                YehorKrivokon Yehor Krivokon
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: