[SUBMARINE-1124] The automatic deletion of notebook does not delete the related PVC and IngressRoute - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.7.0
Component/s: K8s Submitter
Labels:
- pull-request-available

Description

I use newest submarine server to create a notebook called test. Notebook pod can not start by this error with image apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT

Submarine will consistently display the status of the notebook in waiting.

After a long time, submarine will automatically delete the notebook row in the database. I see the corresponding log information in the background.

[INFO ] 2021-12-10 09:23:25,015 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Checked out connection 1073981532 from pool.
Testing connection 1073981532 ...
Connection 1073981532 is GOOD!
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
==>  Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook 
==> Parameters: 
<==    Columns: id, notebook_spec, create_by, create_time, update_by, update_time
<==        Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
<==      Total: 1
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
Testing connection 1073981532 ...
Connection 1073981532 is GOOD!
Returned connection 1073981532 to pool.
[INFO ] 2021-12-10 09:23:25,593 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:25,602 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:25,904 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:25,918 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
[WARN ] 2021-12-10 09:23:27,257 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:27,279 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Checked out connection 1073981532 from pool.
Testing connection 1073981532 ...
[INFO ] 2021-12-10 09:23:27,283 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Connection 1073981532 is GOOD!
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
==>  Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook 
==> Parameters: 
<==    Columns: id, notebook_spec, create_by, create_time, update_by, update_time
<==        Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
<==      Total: 1
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
[WARN ] 2021-12-10 09:23:28,972 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:29,221 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
[INFO ] 2021-12-10 09:23:30,257 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
[INFO ] 2021-12-10 09:23:31,273 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
[INFO ] 2021-12-10 09:23:32,683 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Created connection 1441218336.
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@55e73f20]
==>  Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook 
==> Parameters: 
Created connection 875758614.
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@34330416]
==>  Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook 
==> Parameters: 
Created connection 616832400.
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@24c41d90]
==>  Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook 
==> Parameters: 
Created connection 670559153.
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@27f7ebb1]
==>  Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook 
==> Parameters: 
Created connection 2081921914.
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@7c179b7a]
Testing connection 1073981532 ...
==>  Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook 
==> Parameters: 
Connection 1073981532 is GOOD!
Returned connection 1073981532 to pool.
<==    Columns: id, notebook_spec, create_by, create_time, update_by, update_time
<==        Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
<==      Total: 1
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@34330416]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@34330416]
Testing connection 875758614 ...
<==    Columns: id, notebook_spec, create_by, create_time, update_by, update_time
<==        Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
<==      Total: 1
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@24c41d90]
Connection 875758614 is GOOD!
Returned connection 875758614 to pool.
<==    Columns: id, notebook_spec, create_by, create_time, update_by, update_time
<==        Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
<==      Total: 1
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@27f7ebb1]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@24c41d90]
Testing connection 616832400 ...
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@27f7ebb1]
<==    Columns: id, notebook_spec, create_by, create_time, update_by, update_time
<==        Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
<==      Total: 1
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@7c179b7a]
<==    Columns: id, notebook_spec, create_by, create_time, update_by, update_time
<==        Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09 09:33:27.0, null, 2021-12-09 09:33:27.0
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@7c179b7a]
Connection 616832400 is GOOD!
Returned connection 616832400 to pool.
Testing connection 2081921914 ...
<==      Total: 1
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@55e73f20]
Connection 2081921914 is GOOD!
Returned connection 2081921914 to pool.
Testing connection 670559153 ...
Connection 670559153 is GOOD!
Returned connection 670559153 to pool.
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@55e73f20]
Testing connection 1441218336 ...
<==    Updates: 1
Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145]
<==    Updates: 0
Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f]
Connection 1441218336 is GOOD!
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145]
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145]
Closed connection 1441218336.
Testing connection 1103188293 ...
Connection 1103188293 is GOOD!
Closed connection 1103188293.
Testing connection 307179679 ...
[WARN ] 2021-12-10 09:23:35,350 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
Connection 307179679 is GOOD!
Closed connection 307179679.
[WARN ] 2021-12-10 09:23:35,352 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:38,326 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:38,334 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:38,338 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:38,342 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:40,057 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:40,062 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:42,452 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:42,458 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:42,705 method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON: {"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027 \"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail -1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1; else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2 -d\u0027 \u0027);. Moving forward with env creation and activation.\"; fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off 2m40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off 1m20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off 40s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off 20s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off 10s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off 5m0s restarting failed container\u003dtest pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:42,711 method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is the waiting caused by image pulling
[WARN ] 2021-12-10 09:23:43,055 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[WARN ] 2021-12-10 09:23:43,065 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:43,078 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Checked out connection 1073981532 from pool.
Testing connection 1073981532 ...
Connection 1073981532 is GOOD!
Setting autocommit to false on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
==>  Preparing: select id, notebook_spec, create_by, create_time, update_by, update_time from notebook 
==> Parameters: 
<==      Total: 0
Resetting autocommit to true on JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
Testing connection 1073981532 ...
Connection 1073981532 is GOOD!
Returned connection 1073981532 to pool.
[WARN ] 2021-12-10 09:23:43,083 method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:43,085 method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Checked out connection 875758614 from pool.

I checked codes and found that submarine just delete database row but it does not delete the resource like CRD, PVC and IngressRoute. So that if I recreate with a duplicate name, submarine tell me I can not create this.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

screenshot-1.png
10/Dec/21 02:16
30 kB
cdmikechen
image-2021-12-10-09-56-32-491.png
10/Dec/21 01:56
26 kB
cdmikechen

Issue Links

links to

GitHub Pull Request #855

Activity

People

Assignee:: cdmikechen

Reporter:: cdmikechen

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 10/Dec/21 02:04

Updated:: 04/Jul/22 07:02

Resolved:: 16/Jan/22 09:17