Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-547

Livy kills session after livy.server.session.timeout even if the session is active

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: Server
    • Labels:
      None

      Description

      Livy kills session after livy.server.session.timeout even if the session is active.

      Code that runs more than the livy.server.session.timeout with intermediate sleeps.

      %pyspark 
      import time 
      import datetime 
      import random
      def inside(p):
          x, y = random.random(), random.random()
          return x*x + y*y < 1
      NUM_SAMPLES=10
      count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
                   .filter(inside).count()
      print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
      
      print("waiting for 100 s") 
      time.sleep(100) 
      count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
                   .filter(inside).count()
      print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
      
      print("waiting for 200 s") 
      time.sleep(200) 
      count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
                   .filter(inside).count()
      print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
      
      print("waiting for 300 s1") 
      time.sleep(300)
      count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
                   .filter(inside).count()
      print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
      
      print("waiting for 300 s2") 
      time.sleep(300) 
      count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
                   .filter(inside).count()
      print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
      
      print("waiting for 300 s3") 
      time.sleep(300) 
      count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
                   .filter(inside).count()
      print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
      
      print("waiting for 300 s4") 
      time.sleep(300) 
      count = sc.parallelize(xrange(0, NUM_SAMPLES)) \
                   .filter(inside).count()
      print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
      

      Livy log:

      19/01/07 17:38:59 INFO InteractiveSession: Interactive session 14 created [appid: application_1546711709239_0002, owner: zeppelin-hwc327, proxyUser: Some(admin), state: idle, kind: shared, info: {driverLogUrl=http://hwc327-node3.hogwarts-labs.com:8042/node/containerlogs/container_e18_1546711709239_0002_01_000001/admin, sparkUiUrl=http://hwc327-node2.hogwarts-labs.com:8088/proxy/application_1546711709239_0002/}]
      19/01/07 17:52:46 INFO InteractiveSession: Stopping InteractiveSession 14...
      19/01/07 17:52:56 WARN RSCClient: Exception while waiting for end session reply.
      java.util.concurrent.TimeoutException
              at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:49)
              at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:223)
              at org.apache.livy.server.interactive.InteractiveSession$$anonfun$stopSession$1.apply(InteractiveSession.scala:471)
              at org.apache.livy.server.interactive.InteractiveSession$$anonfun$stopSession$1.apply(InteractiveSession.scala:471)
              at scala.Option.foreach(Option.scala:236)
              at org.apache.livy.server.interactive.InteractiveSession.stopSession(InteractiveSession.scala:471)
              at org.apache.livy.sessions.Session$$anonfun$stop$1.apply$mcV$sp(Session.scala:174)
              at org.apache.livy.sessions.Session$$anonfun$stop$1.apply(Session.scala:171)
              at org.apache.livy.sessions.Session$$anonfun$stop$1.apply(Session.scala:171)
              at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
              at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
              at scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
              at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
              at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
              at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
              at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
      19/01/07 17:52:56 WARN RpcDispatcher: [ClientProtocol] Closing RPC channel with 1 outstanding RPCs.
      19/01/07 17:52:56 WARN InteractiveSession: Failed to stop RSCDriver. Killing it...
      19/01/07 17:52:56 INFO YarnClientImpl: Killed application application_1546711709239_0002
      19/01/07 17:52:56 INFO InteractiveSession: Stopped InteractiveSession 14.
      
      

      Below note indicates that the timeout applies to a inactive session.

      LivyConf.scala
        // How long will an inactive session be gc-ed.
        val SESSION_TIMEOUT = Entry("livy.server.session.timeout", "1h")
      

        Attachments

          Activity

            People

            • Assignee:
              shanyu shanyu zhao
              Reporter:
              Sandeep Nemuri Sandeep Nemuri
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 20m
                1h 20m