Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-278

Statement cancelling improvement

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.3
    • Fix Version/s: None
    • Component/s: REPL
    • Labels:
      None

      Description

      Current implementation of statement cancelling (refer from Zeppelin SparkInterpreter) is not so elegant and threading contention prone:

      1. Using statement to set JobGroup is slow and ugly. Actually this could be done by SparkListener (tracking the mapping between statement and spark jobs). We should consider this with LIVY-272 to offer a sophisticated StatementStateListener to track the progress of statement as well as statement cancelling.
      2. Current statement cancelling can only be worked with Spark jobs. If the statement doesn't include any spark job, the statement cannot be cancelled. We should think of way to handle it.
      3. Current statement cancelling manages complex state transition and threading content. We should refactor to simplify or make the code easy to understand.
      4. If we are still relying on setJobGroupId, move it to Interpreter interface.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jerryshao Saisai Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: