[ZEPPELIN-4988] exception when running zeppelin on the machine where k8s is installed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: interpreter-launcher
• Assignee: Unassigned
• Reporter: zhouli
• Created: 07/Aug/20 08:19
• Updated: 24/Dec/20 03:17
• Resolved: 10/Aug/20 07:55

Description

zeppelin is deployed on a machine which is also a k8s node（zeppelin is running outside of k8s）, and when I run paragraph, the following exception happens:

—

INFO [2020-08-07 10:53:10,776] ({SchedulerFactory2} PluginManager.java[loadInterpreterLauncher]:141) - Loading Interpreter Launcher Plugin: K8sStandardInterpreterLauncher
WARN [2020-08-07 10:53:10,777] ({SchedulerFactory2} PluginManager.java[loadInterpreterLauncher]:152) - *Fail to instantiate InterpreterLauncher from classpath directly:K8sStandardInterpreterLauncher*
java.lang.ClassNotFoundException: org.apache.zeppelin.interpreter.launcher.K8sStandardInterpreterLauncher
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at org.apache.zeppelin.plugin.PluginManager.loadInterpreterLauncher(PluginManager.java:146)
at org.apache.zeppelin.interpreter.InterpreterSetting.createLauncher(InterpreterSetting.java:308)
at org.apache.zeppelin.interpreter.InterpreterSetting.createInterpreterProcess(InterpreterSetting.java:842)
at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:63)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:110)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:163)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:131)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:298)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:431)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:74)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:159)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
WARN [2020-08-07 10:53:10,782] ({SchedulerFactory2} PluginManager.java[loadInterpreterLauncher]:164) - *Fail to instantiate Launcher K8sStandardInterpreterLauncher from plugin pluginDir: /mnt/disk1/zl/app/zeppelin-0.9.0-preview1-bin-all/plugins*
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.zeppelin.plugin.PluginManager.loadInterpreterLauncher(PluginManager.java:161)
at org.apache.zeppelin.interpreter.InterpreterSetting.createLauncher(InterpreterSetting.java:308)
at org.apache.zeppelin.interpreter.InterpreterSetting.createInterpreterProcess(InterpreterSetting.java:842)
at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:63)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:110)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:163)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:131)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:298)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:431)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:74)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:159)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
*Caused by: java.nio.file.NoSuchFileException: /var/run/secrets/kubernetes.io/serviceaccount/namespace*
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215)
at java.base/java.nio.file.Files.newByteChannel(Files.java:370)
at java.base/java.nio.file.Files.newByteChannel(Files.java:421)
at java.base/java.nio.file.Files.readAllBytes(Files.java:3205)
at org.apache.zeppelin.interpreter.launcher.K8sStandardInterpreterLauncher.readFile(K8sStandardInterpreterLauncher.java:174)
at org.apache.zeppelin.interpreter.launcher.K8sStandardInterpreterLauncher.getNamespace(K8sStandardInterpreterLauncher.java:86)
at org.apache.zeppelin.interpreter.launcher.K8sStandardInterpreterLauncher.<init>(K8sStandardInterpreterLauncher.java:51)
... 23 more

—

 

after some research, I found that there is a problem when checking whether zeppelin is running on kubernetes. As shown in the following code snippet,

if /var/run/secrets/kubernetes.io exists, we confirm that zeppelin is running on k8s, however, /var/run/secrets/kubernetes.io not only exists in k8s pod but also in k8s node. so if zeppelin is deployed on a machine which is also a k8s node（zeppelin is running outside of k8s）, we will still confirm that zeppelin is running on k8s, and try to load *K8sStandardInterpreterLauncher*, which is wrong.

instead of **/var/run/secrets/kubernetes.io*, we could use ***/var/run/secrets/kubernetes.io/serviceaccount/namespace** to check whether zeppelin is running on kubernetes, because **/var/run/secrets/kubernetes.io/serviceaccount/namespace** only exists in kubernetes pod.