Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-323

Spark session dies after creation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.3
    • Fix Version/s: 0.4.0
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Description

      I'm trying to create spark session using curl:
      curl -X POST --data '

      {"kind": "spark"}

      ' -H "Content-Type: application/json" localhost:8998/sessions

      Output of curl is about session starting:

      {"id":0,"appId":null,"owner":null,"proxyUser":null,"state":"starting","kind":"spark","appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":[]}
      

      In livy's log (attached) I see messages about spark session creation, but after a while I see a message about spark session successfully created, and immediately it starts shutdown process.

      If I check session status using curl command, it's in DEAD state:

      $ curl http://localhost:8998/sessions | jq .
      {
        "from": 0,
        "total": 1,
        "sessions": [
          {
            "id": 0,
            "appId": "application_1482343367445_0210",
            "owner": null,
            "proxyUser": null,
            "state": "dead",
            "kind": "spark",
            "appInfo": {
              "driverLogUrl": null,
              "sparkUiUrl": "http://ip-10-200-139-129.ec2.internal:20888/proxy/application_1482343367445_0210/"
            },
            "log": []
          }
        ]
      }
      

      In yarn tasks list I can see this session in Finished state and there are no errors or exceptions in driver's log or executor's logs (attached).

      With the same installation, I can successfully start and use pyspack sessions using the same livy server.

        Attachments

          Activity

            People

            • Assignee:
              tc0312 Alex Man
              Reporter:
              shmuma Max Lapan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: