Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2133

All interpreters sometimes throw random Connection refuse errors

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Interpreters
    • Labels:
      None

      Description

      The interpreters sometimes randomly throw connection refused error. When I try to run a paragraph, it initially keeps on printing this:

      DEBUG [2017-02-15 21:26:08,857] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)
      DEBUG [2017-02-15 21:26:09,364] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)
      DEBUG [2017-02-15 21:26:09,871] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)
      DEBUG [2017-02-15 21:26:10,389] ({pool-2-thread-3} RemoteInterpreterUtils.java[checkIfRemoteEndpointAccessible]:53) - Remote endpoint 'localhost:39477' is not accessible (might be initializing): Connection refused (Connection refused)
      

      And when I go to command line I see these zombie processes;

      root@os-r7-zeppelin-we-sec-2 zeppelin# ps aux | grep interpreter
      zeppelin 601 0.0 0.0 113124 1512 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 38959 -l /usr/hdp/current/zeppelin-server/local-repo/2C8CCPY3G
      zeppelin 612 0.0 0.0 113120 844 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 38959 -l /usr/hdp/current/zeppelin-server/local-repo/2C8CCPY3G
      zeppelin 613 0.9 0.2 3427080 42588 ? Sl 06:14 0:00 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 38959
      zeppelin 780 0.0 0.0 113124 1508 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 39741 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
      zeppelin 794 0.0 0.0 113120 844 ? S 06:14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 39741 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
      zeppelin 795 0.9 0.2 3427080 42864 ? Sl 06:14 0:00 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 39741
      zeppelin 1255 0.0 0.0 113124 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 54900 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
      zeppelin 1266 0.0 0.0 113120 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 54900 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
      zeppelin 1267 0.1 0.7 3596948 124976 ? Sl Feb14 2:24 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 54900
      root 1739 0.0 0.0 112648 976 pts/0 S+ 06:15 0:00 grep --color=auto interpreter
      zeppelin 2599 0.0 0.0 113124 240 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 59599 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
      zeppelin 2610 0.0 0.0 113120 240 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 59599 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
      zeppelin 2611 0.1 2.0 3603128 330476 ? Sl Feb14 2:15 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 59599
      zeppelin 2792 0.0 0.0 113124 244 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 34730 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
      zeppelin 2811 0.0 0.0 113120 244 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 34730 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
      zeppelin 2813 0.1 2.0 3598864 336480 ? Sl Feb15 2:12 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 34730
      zeppelin 5441 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 50880 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
      zeppelin 5468 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 50880 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
      zeppelin 5469 0.1 1.9 3600024 322828 ? Sl Feb15 2:10 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 50880
      zeppelin 9968 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 33813 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
      zeppelin 9980 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 33813 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
      zeppelin 9981 0.1 2.0 3598864 338768 ? Sl Feb15 2:13 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 33813
      zeppelin 10263 0.0 0.0 113124 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 43014 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
      zeppelin 10277 0.0 0.0 113120 244 ? S Feb14 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 43014 -l /usr/hdp/current/zeppelin-server/local-repo/2C9XK2KSH
      zeppelin 10279 0.1 2.1 3601948 353012 ? Sl Feb14 2:16 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 43014
      zeppelin 15102 0.0 0.0 113124 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 60490 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
      zeppelin 15113 0.0 0.0 113120 240 ? S Feb15 0:00 /bin/bash /usr/hdp/current/zeppelin-server/bin/interpreter.sh -d /usr/hdp/current/zeppelin-server/interpreter/livy -p 60490 -l /usr/hdp/current/zeppelin-server/local-repo/2C8J12HF5
      zeppelin 15114 0.1 1.7 3600844 285192 ? Sl Feb15 2:11 /usr/lib/jvm/java-openjdk/bin/java -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///usr/hdp/current/zeppelin-server/conf/log4j.properties -Dzeppelin.log.file=/grid/0/log/zeppelin/zeppelin-interpreter-livy-zeppelin-os-r7-zeppelin-we-sec-2.log -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -cp ::/usr/hdp/current/zeppelin-server/interpreter/livy/:/usr/hdp/current/zeppelin-server/lib/interpreter/: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 60490
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                prabhjyotsingh Prabhjyot Singh
                Reporter:
                prabhjyotsingh Prabhjyot Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: