[ZEPPELIN-4983] Packages defined at the interpreter do not work in K8s setup

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.1, 0.10.0
• Component/s: Interpreters, spark
• Assignee: Philipp Dallig
• Reporter: Philipp Dallig
• Created: 03/Aug/20 15:54
• Updated: 16/Apr/21 07:33
• Resolved: 16/Apr/21 07:33

Description

Hi,

I tried the delta extension in my K8s setup.

Seems that packages are not sync to a remote interpreter. How is this solved in other setups?

zjffdu How you solve this in YARN-Cluster`
EDIT: Found YARN solution - Creating a ZIP

moon Do you have an idea to solve this problem in the K8s setup?

Possible Solutions:

Solution	Pro	Con
Zeppelin Sync	(Should) work out of the box	needs new implementation, which is quite complex, because we must sync the content before startup Line, which loads local-repository into classpath may affect other interpreters launcher Dependencies are synchronized each time a new interpreter pod is created
K8s ReadWriteMany Storage (e.g. NFS, GlusterFS, AzureFS)	Zeppelin independent Dependencies are only download once	needs more complex K8s configuration needs a ReadWriteMany Storage doesn't work out of box

Best Regards
Philipp

Issue Links

• is duplicated by: ZEPPELIN-4921 spark.jars not working
• is duplicated by: ZEPPELIN-5013 Depencies not added