[ZEPPELIN-3256] ipython backend: capture when backend ipython process dies

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.7.3, 0.8.0
• Component/s: Core, pySpark, python-interpreter, zeppelin-interpreter
• Assignee: Unassigned
• Reporter: Ruslan Dautkhanov
• Created: 23/Feb/18 04:20
• Updated: 26/Feb/18 16:17

Description

Using `quit()` in the new ipython interpreter backend leads to ipython backend .. exiting, and a new paragraph run gets stuck in 'RUNNING' indefinitely, or at least until pySpark interpreter is restarted.

Two suggestions:

1. Ignore `quit()` calls 
2. More importantly - capture when IPython backend process dies (for this or any other reason) so Spark interpreter would know it has to start a new session, and so it would also not show misleading 'RUNNING' state indefinitely on the front-end to users.

First issue might be easy to fix using something like `def quit(): pass` as soon as ipython process starts.

But again more importantly here to have some generic logic to capture and recognize events when ipython process exits or dies for some reason and pass this information up to Spark interpreter.