[LIVY-421] support multi statements run at the same time

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.4.0
• Fix Version/s: 0.9.0
• Component/s: REPL
• Labels: performance
• Assignee: Unassigned
• Reporter: weibin0516
• Created: 25/Nov/17 06:09
• Updated: 12/Nov/22 22:09

Description

In the v0.4.0-incubating's’ implementation, the number of threads used to execute the statement code is hardcoded to one. When idle resources are sufficient to execute multiple statements, a lot of resources are wasted and blocking statements run. We are considering improvements in one of two ways:

1. 1. Add a configuration in livy.conf to globally control the number of threads in interpreterExecutor for all repl/sessions to execute statements
2. 2 in the session to create a rest api parameter to set the maximum number of statements can be run at the same time the number of statements

hope to discuss this issue, thanks

Issue Links

• is duplicated by: LIVY-428 Does livy support concurrency for centain spark session?