[SUBMARINE-1121] Submarine-Operator Pod CrashLoopBackOff when create a Submarine CRD

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: K8s Submitter
• Labels: pull-request-available
• Assignee: Che-Kuang Chu
• Reporter: cdmikechen
• Created: 09/Dec/21 07:45
• Updated: 15/Oct/22 13:45
• Resolved: 15/Oct/22 13:45

Description

I use helm to create submarine in k8s, when I use Submarine CRD to create server, submarine operator will crash.

Here is my CRD:

apiVersion: submarine.apache.org/v1alpha1
kind: Submarine
metadata: 
  name: example-submarine
spec: 
  version: "0.7.0-SNAPSHOT"
  server: 
    image: "harbor.xxxx.com:5000/apache/submarine:server-0.7.0-SNAPSHOT" # overwrite the image when development
    replicas: 1
  database: 
    image: "harbor.xxxx.com:5000/apache/submarine:database-0.7.0-SNAPSHOT" # overwrite the image when development
    replicas: 1
    storageSize: "1Gi"
    mysqlRootPasswordSecret: "root-pass-secret"
  tensorboard: 
    enabled: true
    storageSize: "10Gi"
  mlflow: 
    enabled: true
    storageSize: "10Gi"
  minio: 
    enabled: true
    storageSize: "10Gi"

Here is submarine operator error:

I1209 07:40:03.828576       1 controller_builder.go:98] Setting up event handlers
I1209 07:40:03.828745       1 controller.go:138] Starting Submarine controller
I1209 07:40:03.828761       1 controller.go:141] Waiting for informer caches to sync
W1209 07:40:03.849789       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W1209 07:40:03.875172       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
I1209 07:40:03.929025       1 controller.go:146] Starting workers
I1209 07:40:03.929058       1 controller.go:152] Started workers
I1209 07:40:03.929097       1 controller.go:235] syncHandler: submarine/example-submarine
I1209 07:40:03.929117       1 submarine_server.go:104] [createSubmarineServer]
I1209 07:40:03.965919       1 submarine_server.go:111] 	Create ServiceAccount:
E1209 07:40:03.966155       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 244 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x150ade0, 0x2185d30)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/runtime/runtime.go:48 +0x86
panic(0x150ade0, 0x2185d30)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).checkSubmarineDependentsReady(0xc00027e400, 0xc000603200, 0x9d, 0xc0002a30e0, 0x11)
	/usr/src/pkg/controller/controller.go:470 +0x159
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).syncHandler(0xc00027e400, 0xc00068ab20, 0x1b, 0xc0004321b0, 0xc0004321b0)
	/usr/src/pkg/controller/controller.go:274 +0x4a5
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).processNextWorkItem.func1(0xc00027e400, 0x14aedc0, 0xc000768940, 0x0, 0x0)
	/usr/src/pkg/controller/controller.go:189 +0xed
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).processNextWorkItem(0xc00027e400, 0x203000)
	/usr/src/pkg/controller/controller.go:199 +0x53
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).runWorker(...)
	/usr/src/pkg/controller/controller.go:163
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000f4c8b0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000f4c8b0, 0x1870980, 0xc000f1f140, 0x1, 0xc00009ade0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000f4c8b0, 0x3b9aca00, 0x0, 0x175ba01, 0xc00009ade0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000f4c8b0, 0x3b9aca00, 0xc00009ade0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90 +0x4d
created by github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).Run
	/usr/src/pkg/controller/controller.go:149 +0x2e5
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x460 pc=0x1355e39]

goroutine 244 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/runtime/runtime.go:55 +0x109
panic(0x150ade0, 0x2185d30)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).checkSubmarineDependentsReady(0xc00027e400, 0xc000603200, 0x9d, 0xc0002a30e0, 0x11)
	/usr/src/pkg/controller/controller.go:470 +0x159
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).syncHandler(0xc00027e400, 0xc00068ab20, 0x1b, 0xc0004321b0, 0xc0004321b0)
	/usr/src/pkg/controller/controller.go:274 +0x4a5
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).processNextWorkItem.func1(0xc00027e400, 0x14aedc0, 0xc000768940, 0x0, 0x0)
	/usr/src/pkg/controller/controller.go:189 +0xed
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).processNextWorkItem(0xc00027e400, 0x203000)
	/usr/src/pkg/controller/controller.go:199 +0x53
github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).runWorker(...)
	/usr/src/pkg/controller/controller.go:163
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000f4c8b0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000f4c8b0, 0x1870980, 0xc000f1f140, 0x1, 0xc00009ade0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000f4c8b0, 0x3b9aca00, 0x0, 0x175ba01, 0xc00009ade0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000f4c8b0, 0x3b9aca00, 0xc00009ade0)
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90 +0x4d
created by github.com/apache/submarine/submarine-cloud-v2/pkg/controller.(*Controller).Run
	/usr/src/pkg/controller/controller.go:149 +0x2e5

Issue Links

• links to: GitHub Pull Request #842