[ZEPPELIN-2358] zeppelin throw PermGen out of memory after a long time running

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 0.6.2
• Component/s: Interpreters
• Labels: performance

Environment

HDP zeppelin 0.6.2
HDP spark 1.6.2
HDP HADOOP 2.6

``` zeppelin-evn

export MASTER=yarn-client
export SPARK_YARN_JAR=spark_jar

export ZEPPELIN_LOG_DIR=zeppelin_log_dir
export ZEPPELIN_PID_DIR=zeppelin_pid_dir

export JAVA_HOME=java64_home

export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=executor_mem -Dspark.executor.instances=executor_instances -Dspark.yarn.queue=spark_queue"

export ZEPPELIN_INTP_MEM="-XX:MaxPermSize=1024m"

export ZEPPELIN_INTP_JAVA_OPTS="-Dhdp.version=2.4.2.0-258 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=1024m"

export SPARK_HOME=/usr/hdp/2.4.2.0-258/spark

export SPARK_SUBMIT_OPTIONS="--driver-memory 7G --executor-memory 16G --jars=/usr/hdp/2.4.2.0-258/zeppelin/interpreter/spark/hanlp-portable-1.3.1.jar"

HADOOP_CONF_DIR.
export HADOOP_CONF_DIR=/etc/hadoop/conf

export PYTHONPATH="${SPARK_HOME}/python:${SPARK_HOME}/python/lib/py4j-0.8.2.1-src.zip"
export SPARK_YARN_USER_ENV="PYTHONPATH=${PYTHONPATH}"

```

• Assignee: Unassigned
• Reporter: poseidon
• Created: 05/Apr/17 11:26
• Updated: 19/Jun/17 02:02
• Resolved: 19/Jun/17 02:02

Description

1、setup zeppelin with ambari
2、config interpreter with a greate mount of PermGen Memory, use share mode for spark interpreter
3、create a notebook in zeppelin
4、run spark code

For some reason I need to use one notebook again and again.

After running like 1000 lines code in notebook.
Zeppelin throw like PermGen Space Out of Memory Exception.

And notebook won't return from running any more.

After check what's in memory with Jprofile, we see result as follow:JUST TOO MUCH LINES.......

Would it be possible if we add some reset things for each interpreter, so we can reuse notebook with out delete and create then again?

=================================================================================
If I change interpreter mode to scope, and delete notebook when i finish some scala code compute. (Because I saw code scope mode will use different interpreter instance and the SparkIMain is not a static field)

I found permgen space is not the problem anymore, but E , O, S0 is 100% taken. And notebook won't return from running any more.

I also check what's in memory .. Fount out no more lines class as I expected, but with ZipArchive???