[ZEPPELIN-5681] Zeppelin image can't use spark on prem

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.10.0
• Fix Version/s: 0.10.0
• Component/s: docker, Interpreters, pySpark, spark, spark-submit, zeppelin-interpreter

Environment

• apache/zeppelin:0.10.0 docker image
• spark 3.1.1 on prem

Language

Python

• Assignee: Unassigned
• Reporter: Dmitry Kravchuk
• Created: 16/Mar/22 06:20
• Updated: 16/Mar/22 09:25
• Resolved: 16/Mar/22 09:25

Description

Hello there!

I have kinda complex task - run zeppelin in docker at on prem cluster and make it possible to use on prem builded spark as a source for spark interpreter.

My docker-compose file looks like this:

version: '3'
services: 
 zeppelin: 
   container_name: zeppelin10
   restart: always
   build: 
      context: /opt/docker/zeppelin/build/.
      dockerfile: /opt/docker/zeppelin/build/Dockerfile
   image: apache/zeppelin:0.10.0
   ports: 
     - "18080:8080"
     - "18443:8443"
     - "18081:8081"
     - "44000-44653:44000-44653"
     - "44655-45000:44655-45000"
   environment: 
     ZEPPELIN_PORT: 8080
     ZEPPELIN_SSL_PORT: 443
     ZEPPELIN_MEM: "-Xms2048m -Xmx2048m -XX:MaxMetaspaceSize=1024m"
     ZEPPELIN_JAVA_OPTS: "-Dspark.executor.memory=8g -Dspark.cores.max=16"
     ZEPPELIN_LOG_DIR: '/opt/zeppelin/logs'
     PATH: '/opt/conda/envs/python_3_with_R/bin:/opt/spark/bin:$PATH'    
   volumes: 
     - s_var-log_bind:/var/log
     - notebook_bind:/opt/zeppelin/notebook
     - /opt/docker/zeppelin/root/opt/zeppelin/conf:/opt/zeppelin/conf
     - /opt/docker/zeppelin/root/opt/packages:/opt/packages
     - /opt/docker/zeppelin/root/opt/delta-core:/opt/delta-core
     - /opt/docker/zeppelin/root/etc/hadoop/conf:/etc/hadoop/conf
     - spark_bind:/opt/spark
     - python_bind:/opt/python-3.7.9
     - /home/zeppelin:/home/zeppelin

volumes: 
  spark_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/opt/spark"
  notebook_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/opt/zeppelin/notebook"
  log_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/opt/zeppelin/logs"
  s_var-log_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/var/log"
  python_bind: 
    driver_opts: 
      type: "none"
      o: "bind"
      device: "/opt/docker/zeppelin/root/opt/python-3.7.9"

Here is my interpreter for on prem spark mounted as volume:

Here is zeppelin output when I'm trying to run default spark interpreter and my own:

So it works in local mode but not in client.

Can you help?