[ZEPPELIN-5447] Spark driver pod cannot create resource "configmaps" under k8s mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.1, 0.10.0
• Component/s: Kubernetes, spark
• Assignee: rickcheng
• Reporter: rickcheng
• Created: 05/Jul/21 09:47
• Updated: 08/Jul/21 06:50
• Resolved: 08/Jul/21 06:50

Description

Description:

When zeppelin is running under k8s mode, it will create the interpreter pod through "k8s/interpreter/100-interpreter-spec.yaml". And if user wants to use the spark interpreter in a k8s cluster with RBAC enabled, the driver pod (i.e., the interpreter pod ) should have the appropriate permissions to list, create, edit and delete pods in the cluster.

 

So in the "k8s/interpreter/100-interpreter-spec.yaml", it defines a role and a rolebinding for the pod to obtain the above permissions. 

 

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: {{zeppelin.k8s.interpreter.pod.name}}
  namespace: {{zeppelin.k8s.namespace}}
  {% if zeppelin.k8s.server.uid is defined %}
  ownerReferences:
  - apiVersion: v1
    controller: false
    blockOwnerDeletion: false
    kind: Pod
    name: {{zeppelin.k8s.server.pod.name}}
    uid: {{zeppelin.k8s.server.uid}}
  {% endif %}
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["create", "get", "update", "list", "delete", "watch" ]

 

However, when I tried to use the spark interpreter under k8s mode, I got the following error (The ip and port of my k8s cluster are replaced. For the complete log, please refer to the attachment log.txt):

 

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://<k8s-ip>:<k8s-port>/api/v1/namespaces/default/configmaps. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps is forbidden: User "system:serviceaccount:default:default" cannot create resource "configmaps" in API group "" in the namespace "default".

 

 

This error seems to be that the service account used cannot correctly create the resource configmap.

In the above defination of the Role in "k8s/interpreter/100-interpreter-spec.yaml", we can see that it only defines the permissions on resource: ["pods", "services"]. 

 

 

Possible solution:

When I add "configmaps" in the resource defination of Role in "k8s/interpreter/100-interpreter-spec.yaml": 

resources: ["pods", "services", "configmaps"]

the spark interpreter can be started correctly. 

 

Reference:

In the official spark documentation (https://spark.apache.org/docs/latest/running-on-kubernetes.html), there is such a description: The service account credentials used by the driver pods must be allowed to create pods, services and configmaps.