Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Invalid
-
None
-
None
-
None
Description
The interpreters sometimes randomly throw connection refused error. When I try to run a paragraph, it initially keeps on printing this:
DEBUG [2017-02-15 21:26:08,857] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused) DEBUG [2017-02-15 21:26:09,364] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused) DEBUG [2017-02-15 21:26:09,871] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused) DEBUG [2017-02-15 21:26:10,389] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)
And when I go to command line I see these zombie processes;
root@os-r7-zeppelin-we-sec-2 zeppelin# ps aux | grep interpreter zeppelin 601 0.0 0.0 113124 1512 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 38959 -l /usr/hdp/current/zeppelin-server/local-repo/2C8CCPY3G zeppelin 612 0.0 0.0 113120 844 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 38959 -l /usr/hdp/current/zeppelin-server/local-repo/2C8CCPY3G zeppelin 613 0.9 0.2 3427080 42588 ? Sl 06:14 0:00 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 38959 zeppelin 780 0.0 0.0 113124 1508 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 39741 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5 zeppelin 794 0.0 0.0 113120 844 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 39741 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5 zeppelin 795 0.9 0.2 3427080 42864 ? Sl 06:14 0:00 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 39741 zeppelin 1255 0.0 0.0 113124 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 54900 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5 zeppelin 1266 0.0 0.0 113120 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 54900 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5 zeppelin 1267 0.1 0.7 3596948 124976 ? Sl Feb14 2:24 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 54900 root 1739 0.0 0.0 112648 976 pts/0 S+ 06:15 0:00 grep --color=auto interpreter zeppelin 2599 0.0 0.0 113124 240 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 59599 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5 zeppelin 2610 0.0 0.0 113120 240 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 59599 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5 zeppelin 2611 0.1 2.0 3603128 330476 ? Sl Feb14 2:15 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 59599 zeppelin 2792 0.0 0.0 113124 244 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 34730 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH zeppelin 2811 0.0 0.0 113120 244 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 34730 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH zeppelin 2813 0.1 2.0 3598864 336480 ? Sl Feb15 2:12 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 34730 zeppelin 5441 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 50880 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH zeppelin 5468 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 50880 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH zeppelin 5469 0.1 1.9 3600024 322828 ? Sl Feb15 2:10 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 50880 zeppelin 9968 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 33813 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH zeppelin 9980 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 33813 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH zeppelin 9981 0.1 2.0 3598864 338768 ? Sl Feb15 2:13 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 33813 zeppelin 10263 0.0 0.0 113124 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 43014 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH zeppelin 10277 0.0 0.0 113120 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 43014 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH zeppelin 10279 0.1 2.1 3601948 353012 ? Sl Feb14 2:16 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 43014 zeppelin 15102 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 60490 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5 zeppelin 15113 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 60490 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5 zeppelin 15114 0.1 1.7 3600844 285192 ? Sl Feb15 2:11 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 60490
Attachments
Issue Links
- blocks
-
ZEPPELIN-2276 Release 0.7.2
-
- Resolved
-
- is related to
-
ZEPPELIN-2037 "Restart" button does not work
-
- Resolved
-
- links to