Major Description
Is there an option in Livy, similar to spark JobServer where we can run statements in an interactive session in parallel?
Apache JobServer has a option to specify the number of requests which we want to run in parallel. Do we have something similar in Livy?
Microsoft Azure HDI support team say - we can run as many statements in a session in parallel as are the number of cores in a headnode in the HDI cluster. But when we tested - it doesn't.
Right now we see bad utilization of resources within a spark session (following interactive session model). Right now we have session per user model. I want to use the session for multiple users, but Livy's model of running only one statement at a time, is acting as a blocker for allowing multiple users to use the same session to run requests.