[LIVY-1003] Interactive session - Setting large value of rsc.server.connect.timeout blocks other tasks

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: RSC
• Assignee: wangdengshan
• Reporter: Tom Chamnongvongse
• Created: 06/Jun/24 14:00
• Updated: 13/Sep/24 08:10
• Resolved: 13/Sep/24 08:10

Description

Problem:

Livy is configured to deploy interactive sessions on YARN with `livy.rsc.server.connect.timeout` configured to a high value. Timeout is increased to allow more time for Livy session to be in YARN `ACCEPTED` state to prevent Livy server from killing the YARN app within the default timeout of 90 seconds.

Until the app is in YARN `RUNNING` state, it takes up a thread in Scala's global execution context - I think due to https://github.com/apache/incubator-livy/blob/v0.8.0-incubating/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474. Creating too many of these sessions that are stuck in `ACCEPTED` state causes other tasks that use that global execution context to be queued up.

How to reproduce:

1. Set `livy.rsc.server.connect.timeout` to something high like 24h.
2. Create enough interactive livy sessions in YARN so that they are queued in ACCEPTED state. The number of sessions that are stuck in ACCEPTED state should be equal to global execution context thread pool size (Runtime.availableProcessors)
3. Try to delete a session using DELETE /sessions/

{sessionId}

and it should hang until one of the sessions is no longer stuck in ACCEPTED state.