Details
Blocker Description
library used: org.apache.hadoop:hadoop-openstack:2.6.0
For long running Swift read operation (e.g., reading a large container), the issued auth token has at most 30 minutes life span from Oracle Storage Service. If the token expired in the middle of the read operation the SwiftRestClient (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701) re-authenticate and acquire a new auth token. However, in the retry request the old, expired token is still used, causing the whole operation to fail.
Because of this bug any meaningful(i.e., long-running) Swift operation is not possible.
Here is a summary of what happened with DEBUG logging turned on:
==========================================================
1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
-------------------------------------------------------------------
2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
token=AccessToken
2. token expiration and re-authentication:
------------------------------------------
2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
298b68dc3b308983f04cb37e8416f13
.
2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
to respond to any of these challenges:
2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
code = 401
2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
Reauthenticating
2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
authentication
2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
Authenticating with Authenticate as tenant 'Storage-paas132' user
'radha.srinivasan@oracle.com' with password of length 9
2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
https://em2.storage.oraclecloud.com/auth/v2.0/tokens
User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
298b68dc3b308983f04cb37e8416f13
.
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
code = 200
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
entry [swift: object-store];
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
swift catalog as swift => object-store
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
[US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c', tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d, expires='2016-09-14T00:26:45Z'}
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
X-Newest: true
X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
298b68dc3b308983f04cb37e8416f13
.
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
code = 204
3. retrying the original request but the auth token used is the old one:
-----------------------------------------------------------------------
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1703 - Retrying
original request
2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
298b68dc3b308983f04cb37e8416f13
Host: em2.storage.oraclecloud.com
.
2016-09-13 19:56:46 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
to respond to any of these challenges: {token=Token}
2016-09-13 19:56:46 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
code = 401
2016-09-13 19:56:46 DEBUG [pool-3-thread-1] SwiftRestClient:1445 - Method GET
on
https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/ failed, status code: 401, status line: HTTP/1.1 401
Unauthorized
2016-09-13 19:56:46 ERROR [pool-3-thread-1] SnapshotJob:244 - Error reading
snapshot
Authentication Failure: Operation not authorized- current access token
=AccessToken
GET
https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/ => 401 : Token not found in cache
at
org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient
.java:1482)
at
org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1
403)
at
org.apache.hadoop.fs.swift.http.SwiftRestClient.findObjects(SwiftRestClient.ja
va:881)
