Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13617

Swift client retrying original request is using expired token after re-authentication

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: fs/swift
    • Labels:
    • Environment:

      Linux EL6

      Description

      library used: org.apache.hadoop:hadoop-openstack:2.6.0

      For long running Swift read operation (e.g., reading a large container), the issued auth token has at most 30 minutes life span from Oracle Storage Service. If the token expired in the middle of the read operation the SwiftRestClient (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701) re-authenticate and acquire a new auth token. However, in the retry request the old, expired token is still used, causing the whole operation to fail.

      Because of this bug any meaningful(i.e., long-running) Swift operation is not possible.

      Here is a summary of what happened with DEBUG logging turned on:
      ==========================================================
      1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
      -------------------------------------------------------------------
      2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
      endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
      objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
      token=AccessToken

      {id='AUTH_tk2dd9d639bbb992089dca008123c3046f', tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493, expires='2016-09-13T23:56:44Z'}

      2. token expiration and re-authentication:
      ------------------------------------------
      2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
      https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
      X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
      User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
      ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
      298b68dc3b308983f04cb37e8416f13
      .
      2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
      to respond to any of these challenges:

      {token=Token}
      2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
      code = 401
      2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
      Reauthenticating
      2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
      authentication
      2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
      Authenticating with Authenticate as tenant 'Storage-paas132' user
      'radha.srinivasan@oracle.com' with password of length 9
      2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
      https://em2.storage.oraclecloud.com/auth/v2.0/tokens
      User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
      ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
      298b68dc3b308983f04cb37e8416f13
      .
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
      code = 200
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
      entry [swift: object-store];
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
      swift catalog as swift => object-store
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
      [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
      endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
      objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
      token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c', tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d, expires='2016-09-14T00:26:45Z'}
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
      authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.

      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
      https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
      X-Newest: true
      X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
      User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
      ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
      298b68dc3b308983f04cb37e8416f13
      .
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
      code = 204

      3. retrying the original request but the auth token used is the old one:
      -----------------------------------------------------------------------
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1703 - Retrying
      original request
      2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
      https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
      X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
      User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
      ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
      298b68dc3b308983f04cb37e8416f13
      Host: em2.storage.oraclecloud.com
      .
      2016-09-13 19:56:46 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
      to respond to any of these challenges: {token=Token}

      2016-09-13 19:56:46 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
      code = 401
      2016-09-13 19:56:46 DEBUG [pool-3-thread-1] SwiftRestClient:1445 - Method GET
      on
      https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/ failed, status code: 401, status line: HTTP/1.1 401
      Unauthorized
      2016-09-13 19:56:46 ERROR [pool-3-thread-1] SnapshotJob:244 - Error reading
      snapshot
      Authentication Failure: Operation not authorized- current access token
      =AccessToken

      {id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c', tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d, expires='2016-09-14T00:26:45Z'}

      GET
      https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/ => 401 : Token not found in cache
      at
      org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient
      .java:1482)
      at
      org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1
      403)
      at
      org.apache.hadoop.fs.swift.http.SwiftRestClient.findObjects(SwiftRestClient.ja
      va:881)

        Attachments

        1. 2016_09_13.stderrout.log
          506 kB
          Steve Yang
        2. HADOOP-13617.patch
          2 kB
          Yulei Li

          Activity

            People

            • Assignee:
              charlse Yulei Li
              Reporter:
              syang Steve Yang

              Dates

              • Created:
                Updated:

                Issue deployment