[ZEPPELIN-2822] Livy Interpreter should have a way to set Livy RSCConf RPC_CLIENT_HANDSHAKE_TIMEOUT

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.2
• Component/s: livy-interpreter
• Assignee: Unassigned
• Reporter: Prabhu Joseph
• Created: 01/Aug/17 08:48
• Updated: 23/Oct/19 16:15

Description

Consider a Node which has NN1 (NameNode) and HiveMetaStore is down but we have HA for both services. Running livy script will create a new session and will wait for ipc.client.connect.timeout (20s) for each jar upload into hdfs

17/07/31 13:59:29 INFO ContextLauncher: 17/07/31 13:59:39 INFO Client: Source and destination file systems are the same. Not copying hdfs://prabhu/hdp/apps/2.6.1.0-129/spark/spark-hdp-assembly.jar
17/07/31 13:59:49 INFO ContextLauncher: 17/07/31 13:59:49 INFO Client: Uploading resource file:/usr/hdp/current/livy-server/rsc-jars/livy-rsc-0.3.0.2.6.1.0-129.jar -> hdfs://prabhu/user/diasmi/.sparkStaging/application_1501501991083_0001/livy-rsc-0.3.0.2.6.1.0-129.jar

and 5 seconds (hive.metastore.client.socket.timeout)

17/07/26 09:09:46 INFO ContextLauncher: 17/07/26 09:09:46 INFO metastore: Trying to connect to metastore with URI thrift://prabhu01:9083
17/07/26 09:09:51 INFO ContextLauncher: 17/07/26 09:09:51 WARN metastore: Failed to connect to the MetaStore Server...
17/07/26 09:09:51 INFO ContextLauncher: 17/07/26 09:09:51 INFO metastore: Trying to connect to metastore with URI thrift://prabhu02:9083
17/07/26 09:09:51 INFO ContextLauncher: 17/07/26 09:09:51 INFO metastore: Connected to metastore.

and finally will fail with timeout with Livy Server Connect Timeout. 90 Seconds is too low for this case. Zeppelin has to have a way for overriding this timeout configuration.

RPC_CLIENT_HANDSHAKE_TIMEOUT("server.connect.timeout", "90s")

17/07/31 14:00:51 ERROR RSCClient: Failed to connect to context.
java.util.concurrent.TimeoutException: Timed out waiting for context to start.
        at com.cloudera.livy.rsc.ContextLauncher.connectTimeout(ContextLauncher.java:133)
        at com.cloudera.livy.rsc.ContextLauncher.access$200(ContextLauncher.java:62)
        at com.cloudera.livy.rsc.ContextLauncher$2.run(ContextLauncher.java:121)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)