Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
-
None
-
Softlayer
Description
When running with persistence to softlayer, Brooklyn works for some time (several days?) but then gets a 401 unauthorized from softlayer.
I suspect jclouds is not renewing the token, but this is TBC.
Additionally, when this happens it affects standbys also so all nodes go into failed status with master unknown and calls to `/v1/server/highAvailability` return a 500 making all consoles unusable.
Stack trace is:
2014-06-23 11:38:36,142 ERROR b.m.h.HighAvailabilityManagerImpl [brooklyn-execmanager-GGyYGAv5-599065]: Problem in HA-poller: org.jclouds.rest.AuthorizationException: request: HEAD https://dal05.objectstorage.service.networklayer.com/v1/AUTH_1234_removed/brooklyn-plane/nodes/GGyYGAv5 HTTP/1.1 failed with response: HTTP/1.1 401 Unauthorized
org.jclouds.rest.AuthorizationException: request: HEAD https://dal05.objectstorage.service.networklayer.com/v1/AUTH_1234_removed/brooklyn-plane/nodes/GGyYGAv5 HTTP/1.1 failed with response: HTTP/1.1 401 Unauthorized
at org.jclouds.openstack.swift.handlers.ParseSwiftErrorFromHttpResponse.handleError(ParseSwiftErrorFromHttpResponse.java:61) ~[swift-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67) ~[jclouds-core-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:180) ~[jclouds-core-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:150) ~[jclouds-core-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:129) ~[jclouds-core-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:95) ~[jclouds-core-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:56) ~[jclouds-core-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) ~[jclouds-core-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) ~[jclouds-core-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at com.sun.proxy.$Proxy61.objectExists(Unknown Source) ~[na:na]
at org.jclouds.openstack.swift.blobstore.SwiftBlobStore.blobExists(SwiftBlobStore.java:163) ~[swift-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
at brooklyn.entity.rebind.persister.jclouds.JcloudsStoreObjectAccessor.exists(JcloudsStoreObjectAccessor.java:33) ~[brooklyn-locations-jclouds-0.7.0-SNAPSHOT.jar:na]
at brooklyn.entity.rebind.persister.StoreObjectAccessorLocking.exists(StoreObjectAccessorLocking.java:59) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
at brooklyn.management.ha.ManagementPlaneSyncRecordPersisterToObjectStore.persist(ManagementPlaneSyncRecordPersisterToObjectStore.java:251) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
at brooklyn.management.ha.ManagementPlaneSyncRecordPersisterToObjectStore.delta(ManagementPlaneSyncRecordPersisterToObjectStore.java:206) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
at brooklyn.management.ha.HighAvailabilityManagerImpl.publishHealth(HighAvailabilityManagerImpl.java:282) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
at brooklyn.management.ha.HighAvailabilityManagerImpl.publishAndCheck(HighAvailabilityManagerImpl.java:270) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
at brooklyn.management.ha.HighAvailabilityManagerImpl$2.run(HighAvailabilityManagerImpl.java:245) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_27]
at brooklyn.util.task.BasicExecutionManager$2$1.call(BasicExecutionManager.java:300) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
at brooklyn.util.task.BasicExecutionManager$3.call(BasicExecutionManager.java:353) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_27]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_27]
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://dal05.objectstorage.service.networklayer.com/v1/AUTH_1234_removed/brooklyn-plane/nodes/GGyYGAv5 HTTP/1.1 failed with response: HTTP/1.1 401 Unauthorized
at org.jclouds.openstack.swift.handlers.ParseSwiftErrorFromHttpResponse.handleError(ParseSwiftErrorFromHttpResponse.java:55) ~[swift-1.7.4-ea-brooklyn.1.jar:1.7.4-ea-brooklyn.1]
... 25 common frames omitted