[ZEPPELIN-3860] Zeppelin Broken pipe due to non terminating RemoteInterpreterProcess

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.7.3

Environment

HDP 2.6

• External issue ID: ZEPPELIN-1832
• Assignee: Unassigned
• Reporter: Kislay Kumar
• Created: 12/Nov/18 02:38
• Updated: 12/Nov/18 05:17

Description

This issue may be related to : ZEPPELIN-1832 (it had temporary fix)

 

Broken pipe exception is repeatedly coming for Zeppelin. It starts with SocketException. 

 

Some logs are as under: 

WARNING: A servlet request to the URI https://xxyy:9995/api/login contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

Exception in thread "Thread-815" Exception in thread "Thread-809" java.util.ConcurrentModificationException

        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)

        at java.util.HashMap$ValueIterator.next(HashMap.java:1466)

        at org.apache.zeppelin.interpreter.InterpreterSetting.shutdownAndRemoveAllInterpreterGroups(InterpreterSetting.java:263)

        at org.apache.zeppelin.interpreter.InterpreterSettingManager$7.run(InterpreterSettingManager.java:1083)

java.util.ConcurrentModificationException

        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)

        at java.util.HashMap$ValueIterator.next(HashMap.java:1466)

        at org.apache.zeppelin.interpreter.InterpreterSetting.shutdownAndRemoveAllInterpreterGroups(InterpreterSetting.java:263)

        at org.apache.zeppelin.interpreter.InterpreterSettingManager$7.run(InterpreterSettingManager.java:1083)

Zeppelin is restarting

 

More logs from 

INFO [2018-11-09 00:00:01,334] ({Thread-825} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException

-----------------------------------------

DEBUG [2018-11-09 00:01:31,742] ({Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@39aaeb44{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
DEBUG [2018-11-09 00:01:31,743] ({qtp240166646-959} AbstractConnection.java[onFillInterestedFailed]:168) - HttpConnection@6b820a35[IDLE,DecryptedEndPoint@379cfbb3

{/10.111.1.138:60502<->9995,Open,in,out,-,-,30002/30000,HttpConnection}

->SelectChannelEndPoint@7b0e8d49

{/10.111.1.138:60502<->9995,Open,in,out,-,-,1/30000,SslConnection} {io=1,kio=1,kro=1}

][p=HttpParser

{s=CLOSED,0 of -1},g=HttpGenerator\\{s=START},c=HttpChannelOverHttp@5456b387
{r=9,c=false,a=IDLE,uri=}] onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
DEBUG [2018-11-09 00:01:31,743] ({Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@cbde656{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
DEBUG [2018-11-09 00:01:31,744] ({qtp240166646-975} AbstractConnection.java[onFillInterestedFailed]:168) - HttpConnection@671a181f[IDLE,DecryptedEndPoint@4bb91807{/10.111.1.138:60503<->9995,Open,in,out,-,-,30001/30000,HttpConnection}->SelectChannelEndPoint@efc362c{/10.111.1.138:60503<->9995,Open,in,out,-,-,0/30000,SslConnection}{io=1,kio=1,kro=1}

][p=HttpParser
{s=CLOSED,0 of -1}

,g=HttpGenerator

{s=START}

,c=HttpChannelOverHttp@524d84e3

{r=8,c=false,a=IDLE,uri=}

] onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
DEBUG [2018-11-09 00:01:32,477] ({Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@4c568d5d{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

---------------------------

INFO [2018-11-09 01:17:08,547] ({Thread-824} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException
org.apache.thrift.transport.TTransportException
org.apache.thrift.transport.TTransportException
ERROR [2018-11-09 01:17:08,609] ({qtp240166646-1125 - /api/helium/suggest/2D2V1VPF3/20171128-114541_374495247} ResourcePoolUtils.java[getAllResourcesExcept]:64) - org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
org.apache.thrift.transport.TTransportException
DEBUG [2018-11-09 01:17:09,424] ({qtp240166646-949} WriteFlusher.java[onFail]:460) - failed: WriteFlusher@248e2b43{PENDING} java.nio.channels.ClosedChannelException
INFO [2018-11-09 01:17:11,007] ({Thread-1017} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException

--------------------------------------------------------------

 DEBUG [2018-11-09 00:00:01,331] ({qtp240166646-18-selector-ServerConnectorManager@223f3642/1} SelectorManager.java[select]:599) - Selector loop waiting on select
INFO [2018-11-09 00:00:01,334] ({Thread-825} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
DEBUG [2018-11-09 00:00:01,334] ({Thread-825} RemoteInterpreterProcess.java[dereference]:135) -
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_shutdown(RemoteInterpreterService.java:386)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.shutdown(RemoteInterpreterService.java:374)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.dereference(RemoteInterpreterProcess.java:130)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.close(RemoteInterpreter.java:326)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.close(LazyOpenInterpreter.java:84)
at org.apache.zeppelin.interpreter.InterpreterGroup$1.run(InterpreterGroup.java:198)
INFO [2018-11-09 00:00:01,338] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:191) - Interpreter process exited 0