[SUBMARINE-1229] Minio initContainer pod (mlflow and server) not working properly in openshift

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Component/s: Cloud-native Deployment
• Labels: pull-request-available
• Assignee: cdmikechen
• Reporter: cdmikechen
• Created: 20/Mar/22 07:29
• Updated: 05/Apr/22 03:16
• Resolved: 05/Apr/22 03:16

Description

Minio initContainer pod always shows that it cannot end correctly. I've seen the log in the pod and found the following errors:

mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.
mc: <ERROR> Unable to save new mc config. mkdir /.mc: permission denied.

I checked resource yaml and found that openshift add a runasuser in deployment:

initContainers:
  - resources: {}
    terminationMessagePath: /dev/termination-log
    name: submarine-server-initcontainer
    command:
      - /bin/bash
      - '-c'
      - >-
        cnt=0; while ! /bin/bash -c 'mc config host add minio
        http://submarine-minio-service:9000 submarine_minio submarine_minio'
        2>&1; do sleep 15; ((cnt=cnt+1)); if [ $cnt -eq 80 ];then echo 'ERROR:
        wait too long for minio pod'; exit 1; fi; done; if /bin/bash -c 'mc ls
        minio/submarine' >/dev/null 2>&1; then echo 'Bucket minio/submarine
        already exists, skipping creation.'; else /bin/bash -c 'mc mb
        minio/submarine'; fi;
    securityContext:
      capabilities:
        drop:
          - KILL
          - MKNOD
          - SETGID
          - SETUID
      runAsUser: 1000790000
    imagePullPolicy: Always
    volumeMounts:
      - name: submarine-server-token-bnj4d
        readOnly: true
        mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    terminationMessagePolicy: File
    image: 'minio/mc'

Openshift will add runAsUser with a custom uid under securityContext when we didn't add RunAsAny to serviceaccount.

oc adm policy add-scc-to-user anyuid -z submarine-server -n submarine
oc adm policy add-scc-to-user anyuid -z default -n submarine

Issue Links

• links to: GitHub Pull Request #918