Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2133

All interpreters sometimes throw random Connection refuse errors

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Invalid
    • None
    • None
    • Interpreters
    • None

    Description

      The interpreters sometimes randomly throw connection refused error. When I try to run a paragraph, it initially keeps on printing this:

      DEBUG [2017-02-15 21:26:08,857] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)
DEBUG [2017-02-15 21:26:09,364] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)
DEBUG [2017-02-15 21:26:09,871] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)
DEBUG [2017-02-15 21:26:10,389] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)

      And when I go to command line I see these zombie processes;

      root@os-r7-zeppelin-we-sec-2 zeppelin# ps aux | grep interpreter
zeppelin 601 0.0 0.0 113124 1512 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 38959 -l /usr/hdp/current/zeppelin-server/local-repo/2C8CCPY3G
zeppelin 612 0.0 0.0 113120 844 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 38959 -l /usr/hdp/current/zeppelin-server/local-repo/2C8CCPY3G
zeppelin 613 0.9 0.2 3427080 42588 ? Sl 06:14 0:00 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 38959
zeppelin 780 0.0 0.0 113124 1508 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 39741 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
zeppelin 794 0.0 0.0 113120 844 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 39741 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
zeppelin 795 0.9 0.2 3427080 42864 ? Sl 06:14 0:00 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 39741
zeppelin 1255 0.0 0.0 113124 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 54900 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
zeppelin 1266 0.0 0.0 113120 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 54900 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
zeppelin 1267 0.1 0.7 3596948 124976 ? Sl Feb14 2:24 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 54900
root 1739 0.0 0.0 112648 976 pts/0 S+ 06:15 0:00 grep --color=auto interpreter
zeppelin 2599 0.0 0.0 113124 240 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 59599 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
zeppelin 2610 0.0 0.0 113120 240 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 59599 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
zeppelin 2611 0.1 2.0 3603128 330476 ? Sl Feb14 2:15 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 59599
zeppelin 2792 0.0 0.0 113124 244 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 34730 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
zeppelin 2811 0.0 0.0 113120 244 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 34730 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
zeppelin 2813 0.1 2.0 3598864 336480 ? Sl Feb15 2:12 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 34730
zeppelin 5441 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 50880 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
zeppelin 5468 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 50880 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
zeppelin 5469 0.1 1.9 3600024 322828 ? Sl Feb15 2:10 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 50880
zeppelin 9968 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 33813 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
zeppelin 9980 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 33813 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
zeppelin 9981 0.1 2.0 3598864 338768 ? Sl Feb15 2:13 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 33813
zeppelin 10263 0.0 0.0 113124 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 43014 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
zeppelin 10277 0.0 0.0 113120 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 43014 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
zeppelin 10279 0.1 2.1 3601948 353012 ? Sl Feb14 2:16 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 43014
zeppelin 15102 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 60490 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
zeppelin 15113 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 60490 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
zeppelin 15114 0.1 1.7 3600844 285192 ? Sl Feb15 2:11 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 60490

      Attachments

        Issue Links

          blocks

          Task - A task that needs to be done. ZEPPELIN-2276 Release 0.7.2

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. ZEPPELIN-2037 "Restart" button does not work

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #2034

          Activity

            People

              prabhjyotsingh Prabhjyot Singh
              prabhjyotsingh Prabhjyot Singh
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: