Critical Description
Currently livy GET /Sessions doesnt return fields like driver-executor memory, spark conf, etc. Ideally a session response should return all the values set in session request api call(POST /sessions).
The current response for GET /sessions or GET /sessions/{id} is (ref: https://livy.incubator.apache.org/docs/latest/rest-api.html).
| Name | Description | Type |
|---|---|---|
| id | The session id | int |
| appId | The application id of this session | string |
| owner | Remote user who submitted this session | string |
| proxyUser | User to impersonate when running | string |
| kind | Session kind (spark, pyspark, sparkr, or sql) | session kind |
| log | The log lines | list of strings |
| state | The session state | string |
| appInfo | The detailed application info | Map of key=val |
However the session requests for additional spark related configurations like
- jars
- pyFiles
- files
- driverMemory
- driverCores
- executorMemory
- executorCores
- numExecutors
- archives
- queue
- name
- conf
- heartbeatTimeoutInSecond
The API should return these fields with the response as well.
