[ZEPPELIN-3505] IPython interpreter: ERROR:tornado.general:Uncaught exception in ZMQStream callback

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.8.0, 0.8.1, 0.9.0
• Component/s: Interpreters, pySpark, python-interpreter, zeppelin-interpreter
• Assignee: Unassigned
• Reporter: Ruslan Dautkhanov
• Created: 26/May/18 17:00
• Updated: 09/Aug/18 20:58

Description

Getting following exceptions in IPython interpreter..

ERROR:tornado.general:Uncaught exception in ZMQStream callback 
ValueError: signal only works in main thread
ERROR:tornado.general:Uncaught exception in zmqstream callback
ERROR:tornado.application:Exception in callback <functools.partial object at 0x7fa07f08e998>
...
raise RuntimeError("IOLoop is already running")

Complete list of exceptions -

DEBUG [2018-05-26 10:28:29,453] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: Traceback (most recent call last):
DEBUG [2018-05-26 10:28:29,453] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
DEBUG [2018-05-26 10:28:29,454] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     "__main__", fname, loader, pkg_name)
DEBUG [2018-05-26 10:28:29,454] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/runpy.py", line 72, in _run_code
DEBUG [2018-05-26 10:28:29,454] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     exec code in run_globals
DEBUG [2018-05-26 10:28:29,454] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda-4.4.0/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
DEBUG [2018-05-26 10:28:29,454] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     app.launch_new_instance()
DEBUG [2018-05-26 10:28:29,454] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     app.start()
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: ERROR:tornado.general:Uncaught exception in ZMQStream callback
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: Traceback (most recent call last):
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     callback(*args, **kwargs)
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tornado/stack_context.py", line 276, in null_wrapper
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     return fn(*args, **kwargs)
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     return self.dispatch_shell(stream, msg)
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self.pre_handler_hook()
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 248, in pre_handler_hook
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self.saved_sigint_handler = signal(SIGINT, default_int_handler)
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: ValueError: signal only works in main thread
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 477, in start
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: ERROR:tornado.general:Uncaught exception in zmqstream callback
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: Traceback (most recent call last):
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
DEBUG [2018-05-26 10:28:29,455] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self._handle_recv()
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self._run_callback(callback, msg)
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     callback(*args, **kwargs)
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tornado/stack_context.py", line 276, in null_wrapper
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     return fn(*args, **kwargs)
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     return self.dispatch_shell(stream, msg)
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self.pre_handler_hook()
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 248, in pre_handler_hook
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self.saved_sigint_handler = signal(SIGINT, default_int_handler)
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: ValueError: signal only works in main thread
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     ioloop.IOLoop.instance().start()
DEBUG [2018-05-26 10:28:29,456] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tornado/ioloop.py", line 928, in start
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: ERROR:tornado.application:Exception in callback <functools.partial object at 0x7fa07f08e998>
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: Traceback (most recent call last):
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tornado/ioloop.py", line 759, in _run_callback
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     ret = callback()
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tornado/stack_context.py", line 276, in null_wrapper
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     return fn(*args, **kwargs)
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 536, in <lambda>
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self._handle_recv()
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self._run_callback(callback, msg)
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     callback(*args, **kwargs)
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/tornado/stack_context.py", line 276, in null_wrapper
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     return fn(*args, **kwargs)
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     return self.dispatch_shell(stream, msg)
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self.pre_handler_hook()
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 248, in pre_handler_hook
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     self.saved_sigint_handler = signal(SIGINT, default_int_handler)
DEBUG [2018-05-26 10:28:29,457] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: ValueError: signal only works in main thread
DEBUG [2018-05-26 10:28:29,458] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output:     raise RuntimeError("IOLoop is already running")
DEBUG [2018-05-26 10:28:29,458] ({Exec Stream Pumper} IPythonInterpreter.java[processLine]:411) - Process Output: RuntimeError: IOLoop is already running

 

We did upgrade Anaconda Python from 4.3.1 to 4.4.0 and it may have upgraded certain Python libraries and it broke some things in Zeppelin IPython interpreter?

Is there is a particular list of Python modules that have to be installed documented somewhere? Their versions?

Issue Links

• links to: GitHub Pull Request #3084