Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-701

createNodesInGroup fails with IllegalStateException if we hit CPU quota on GCE

    XMLWordPrintableJSON

Details

    Description

      When we call createNodesInGroup for a Google Compute Engine project that has come up against its CPU quota, we see this exception:

      2014-09-02 15:55:03,640 ERROR [pool-3-thread-4] j.compute [SLF4JLogger.java:93] createNodesInGroup(nothing), completed: 0/2, errors: 1, rate: 6586ms/op
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: operation failed. Http Error Code: 403 HttpError: FORBIDDEN
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-17.0.jar:na]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-17.0.jar:na]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-17.0.jar:na]
	at org.jclouds.concurrent.FutureIterables$1.run(FutureIterables.java:123) ~[jclouds-core-1.8.0.jar:1.8.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
Caused by: java.lang.IllegalStateException: operation failed. Http Error Code: 403 HttpError: FORBIDDEN
	at org.jclouds.googlecomputeengine.compute.GoogleComputeEngineServiceAdapter.waitOperationDone(GoogleComputeEngineServiceAdapter.java:434) ~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
	at org.jclouds.googlecomputeengine.compute.GoogleComputeEngineServiceAdapter.createNodeWithGroupEncodedIntoName(GoogleComputeEngineServiceAdapter.java:185) ~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
	at org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.createNodeWithGroupEncodedIntoName(AdaptingComputeServiceStrategies.java:195) ~[jclouds-compute-1.8.0.jar:1.8.0]
	at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:79) ~[jclouds-compute-1.8.0.jar:1.8.0]
	at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:63) ~[jclouds-compute-1.8.0.jar:1.8.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
	at java.lang.Thread.getStackTrace(Thread.java:1589) [na:1.7.0_60]
	at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:167) ~[jclouds-compute-1.8.0.jar:1.8.0]
	at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122) ~[jclouds-compute-1.8.0.jar:1.8.0]
	at org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:127) ~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:210) ~[jclouds-compute-1.8.0.jar:1.8.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_60]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_60]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_60]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_60]
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) ~[guice-3.0.jar:na]
	at com.sun.proxy.$Proxy124.createNodesInGroup(Unknown Source) ~[na:na]
	at mesosphere.poseidon.providers.Google.createNodesFromTemplate(Google.scala:58) ~[classes/:na]

      This is inconsistent with the error we get when the network quota is exceeded, which is:

      2014-08-07 22:01:37,109 WARN [ApiWorker-2] m.p.p.Google [Provider.scala:142] development-24-88d79664.dcc.io // Retrying after:
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Could not create network, operation failedOperation{kind=compute#operation, id=3290207288552397268, name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860, selfLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/operations/operation-1407448895523-500113b7508b8-d2ab879d-4773f860, name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860, targetLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/networks/jclouds-development-24, status=DONE, user=tobi@mesosphere.io, progress=100, insertTime=Thu Aug 07 22:01:35 UTC 2014, startTime=Thu Aug 07 22:01:35 UTC 2014, endTime=Thu Aug 07 22:01:36 UTC 2014, httpError={statusCode=403, message=FORBIDDEN, headers={}}, operationType=insert, errors=[Error{code=QUOTA_EXCEEDED, message=Quota 'NETWORKS' exceeded.  Limit: 5.0}]}
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache$LocalLoadingCache.apply(LocalCache.java:4830) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.getOrCreateNetwork(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:137) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:215) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_55]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_55]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_55]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55]
    at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.sun.proxy.$Proxy124.createNodesInGroup(Unknown Source) ~[na:na]
    at mesosphere.poseidon.providers.Google$$anonfun$1$$anonfun$apply$1.apply(Google.scala:58) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at mesosphere.poseidon.providers.Google$$anonfun$1$$anonfun$apply$1.apply(Google.scala:55) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
Caused by: java.lang.IllegalStateException: Could not create network, operation failedOperation{kind=compute#operation, id=3290207288552397268, name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860, selfLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/operations/operation-1407448895523-500113b7508b8-d2ab879d-4773f860, name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860, targetLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/networks/jclouds-development-24, status=DONE, user=tobi@mesosphere.io, progress=100, insertTime=Thu Aug 07 22:01:35 UTC 2014, startTime=Thu Aug 07 22:01:35 UTC 2014, endTime=Thu Aug 07 22:01:36 UTC 2014, httpError={statusCode=403, message=FORBIDDEN, headers={}}, operationType=insert, errors=[Error{code=QUOTA_EXCEEDED, message=Quota 'NETWORKS' exceeded.  Limit: 5.0}]}
    at com.google.common.base.Preconditions.checkState(Preconditions.java:150) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at org.jclouds.googlecomputeengine.functions.CreateNetworkIfNeeded.apply(CreateNetworkIfNeeded.java:95) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at org.jclouds.googlecomputeengine.functions.CreateNetworkIfNeeded.apply(CreateNetworkIfNeeded.java:46) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at org.jclouds.googlecomputeengine.compute.loaders.FindNetworkOrCreate.load(FindNetworkOrCreate.java:59) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at org.jclouds.googlecomputeengine.compute.loaders.FindNetworkOrCreate.load(FindNetworkOrCreate.java:36) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
    ... 21 common frames omitted

      Attachments

        Activity

          People

            Unassigned Unassigned
            ssk2hd Sunil Shah
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: