[ZEPPELIN-3542] Stop a pyspark paragraph cancels all the pyspark jobs of other users

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8.0
• Component/s: pySpark

Environment

This happen when the spark context is shared between users (scoped).

• Assignee: Unassigned
• Reporter: Jhon Cardenas
• Created: 14/Jun/18 19:04
• Updated: 14/Jun/18 19:14

Description

Stop a pyspark paragraph cancels all the pyspark jobs of other users.

Cancel button in pyspark paragraph cancels Spark jobs for all users.

This happen when the spark context is shared between users (scoped).

It seems to be related with the solution for the task ZEPPELIN-2075.

This solution is causing that when one particular user cancels his py-spark job, the py-spark jobs from all the users are being canceled !!.

When a py-spark job is cancelled, the method PySparkInterpreter interrupt() is invoked, and then the SIGINT event is called, causing that all the jobs in the same spark context be cancelled:

context.py:

# create a signal handler which would be invoked on receiving SIGINT
def signal_handler(signal, frame):
    self.cancelAllJobs()
    raise KeyboardInterrupt()

Issue Links

• is caused by: ZEPPELIN-2075 Can't stop infinite `while` statement in pyspark Interpreter.