Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-10092

Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header



    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 1.7.3, 1.8.2, 1.9.1, 1.10.0
    • None
    • None


      This problem was encountered when trying to specify container image nvcr.io/nvidia/tensorflow:19.12-tf1-py3

      When initiating Docker Registry authentication (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI fetcher receives 'WWW-Authenticate' header without 'service' and 'scope' params, and fails here:

      This is an example of an unsuccessful request made by Mesos:

      curl -s -S -L -i --raw --http1.1 -H "Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws" -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3
      HTTP/1.1 401 Unauthorized
      Content-Type: text/html
      Date: Wed, 22 Jan 2020 19:01:57 GMT
      Server: nginx/1.14.2
      Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push"
      Content-Length: 195
      Connection: keep-alive
      <head><title>401 Authorization Required</title></head>
      <body bgcolor="white">
      <center><h1>401 Authorization Required</h1></center>

      At the same time, docker is perfectly capable of pulling this image.

      Note that the document "Token Authentication Specification" (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos implementation is based, is vague on the issue of registries that do not provide 'scope'/'service' in WWW-Authenticate header.

      What Docker does differently (at the very least, in the case of nvcr.io):

      It sends the initial request not to the maniferst/blob URI, but to the repository root URI (http:://nvcr.io/v2 in this case):

      GET /v2/ HTTP/1.1
      Host: nvcr.io
      User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd kernel/4.15.0-60-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.7 \(linux\))

      To this, it receives response with a "realm" that contains no query arguments:

      HTTP/1.1 401 Unauthorized
      Connection: close
      Content-Length: 195
      Content-Type: text/html
      Date: Wed, 29 Jan 2020 12:22:43 GMT
      Server: nginx/1.14.2
      Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth

      Then, it composes the scope using the image ref and a hardcoded "pull" action:
      (in a full accordance with this spec: https://docs.docker.com/registry/spec/auth/scope/)

      and sends the following request to https://nvcr.io/proxy_auth :

      GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTTP/1.1
      Host: nvcr.io
      User-Agent: Go-http-client/1.1

      (Note that 'push' is absent from the scope)




            asekretenko Andrei Sekretenko
            asekretenko Andrei Sekretenko
            0 Vote for this issue
            1 Start watching this issue