Major Description
From the docs:
Starting with version 0.5.0-incubating, each session can support all four Scala, Python and R interpreters with newly added SQL interpreter. The {{kind}}field in session creation is no longer required, instead users should specify code kind (spark, pyspark, sparkr or sql) during statement submission.
To be compatible with previous versions, users can still specify kind in session creation, while ignoring kind in statement submission. Livy will then use this session kind as default kind for all the submitted statements.
1. I've found a 5th value for Session kind (only 4 are documented: https://livy.incubator.apache.org/docs/latest/rest-api.html#session-kind )
2. In 'shared' case, "users cannot still specify kind in session creation, while ignoring kind" due to the error that is received demonstrated below.
harsch@mint64 ~ $ curl -s -X POST --data '{}' -H "Content-Type: application/json" localhost:8998/sessions | python -m json.tool { "appId": null, "appInfo": { "driverLogUrl": null, "sparkUiUrl": null }, "id": 0, "kind": "shared", "log": [ "stdout: ", "\nstderr: " ], "owner": null, "proxyUser": null, "state": "starting" }
Executing this:
curl -s -X POST --data '{"code":"1 + 1"}' -H "Content-Type: application/json" localhost:8998/sessions/0/statements| python -m json.tool
will produce this in the logs:
Caused by: org.apache.livy.rsc.rpc.RpcException: java.lang.IllegalArgumentException: Code type should be specified if session kind is shared
I do not know how a 'shared' session kind is different from "spark" session kind, or I would suggest some further documentation that would distinguish them. Perhaps one of the dev team could come up with some useful documentation in this regard.