[BROOKLYN-424] Web-server hangs after several days - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.11.0
Labels:
None

Description

Using Brooklyn 0.10.0-SNAPSHOT from (approximately) 13-09-16...

The web-server becomes unresponsive. Refreshing the page doesn't help.

There are (probably) approx 100 catalog items in the persisted state, but no apps deployed.

Looking at the jstack output, there are 16 threads (all with names matching "brooklyn-jetty-server-8081-qtp97076093-.*") that are parked on the same groovy LockableObject (0x00000007030f2d50).

Most of the stacktraces looks something like:

"brooklyn-jetty-server-8081-qtp97076093-17830" prio=10 tid=0x00007f430c5db800 nid=0x1d4d waiting on condition [0x00007f42c5cd8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007030f2d50> (a org.codehaus.groovy.util.LockableObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:34)
        at org.codehaus.groovy.reflection.ClassInfo.lock(ClassInfo.java:327)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:239)
        at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:255)
        at groovy.lang.MetaClassImpl.pickStaticMethod(MetaClassImpl.java:1511)
        at groovy.lang.MetaClassImpl.retrieveStaticMethod(MetaClassImpl.java:1401)
        at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1435)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:875)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:166)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.isCase(ScriptBytecodeAdapter.java:657)
        at org.apache.brooklyn.util.groovy.GroovyJavaMethods.fix(GroovyJavaMethods.java:176)
        at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
        at org.apache.brooklyn.util.core.flags.FlagUtils.getFlagConfigKeyRecord(FlagUtils.java:277)
        at org.apache.brooklyn.util.core.flags.FlagUtils.findAllFlagsAndConfigKeys(FlagUtils.java:250)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.findAllFlagsAndConfigKeyValues(BrooklynComponentTemplateResolver.java:378)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.configureEntityConfig(BrooklynComponentTemplateResolver.java:275)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.decorateSpec(BrooklynComponentTemplateResolver.java:255)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.populateSpec(BrooklynComponentTemplateResolver.java:246)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.resolveSpec(BrooklynComponentTemplateResolver.java:185)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.buildTemplateServicesAsSpecs(BrooklynAssemblyTemplateInstantiator.java:114)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.createServiceSpecs(BrooklynAssemblyTemplateInstantiator.java:73)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynAssemblyTemplateInstantiator.createApplicationSpec(BrooklynAssemblyTemplateInstantiator.java:90)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.CampResolver.createEntitySpecFromServicesBlock(CampResolver.java:145)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.CampResolver.createSpecFromFull(CampResolver.java:109)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.CampToSpecTransformer.createCatalogSpec(CampToSpecTransformer.java:102)
        at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog$1.apply(BasicBrooklynCatalog.java:331)
        at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog$1.apply(BasicBrooklynCatalog.java:328)
        at org.apache.brooklyn.core.plan.PlanToSpecFactory.attemptWithLoaders(PlanToSpecFactory.java:126)
        at org.apache.brooklyn.core.plan.PlanToSpecFactory.attemptWithLoaders(PlanToSpecFactory.java:118)
        at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.internalCreateSpecLegacy(BasicBrooklynCatalog.java:328)
        at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.createSpec(BasicBrooklynCatalog.java:310)
        at org.apache.brooklyn.rest.transform.CatalogTransformer.catalogEntitySummary(CatalogTransformer.java:81)
        at org.apache.brooklyn.rest.transform.CatalogTransformer.catalogItemSummary(CatalogTransformer.java:117)
        at org.apache.brooklyn.rest.resources.CatalogResource$1.apply(CatalogResource.java:91)
        at org.apache.brooklyn.rest.resources.CatalogResource$1.apply(CatalogResource.java:88)
        at com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:582)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContentsUsing(CollectionSerializer.java:139)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:87)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:23)
        at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:250)
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:604)
        at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:648)
        at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1378)
        at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:235)
        at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:118)
        at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:81)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
        - locked <0x00000007dec2d290> (a org.apache.cxf.phase.PhaseInterceptorChain)
        at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
        - locked <0x00000007dec17dc0> (a org.apache.cxf.phase.PhaseInterceptorChain)
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:293)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:217)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
        at org.apache.brooklyn.rest.filter.LoggingFilter.doFilter(LoggingFilter.java:91)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.apache.brooklyn.rest.filter.RequestTaggingFilter.doFilter(RequestTaggingFilter.java:51)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)

And one looks like:

"brooklyn-jetty-server-8081-qtp97076093-16662" prio=10 tid=0x00007f430c0da000 nid=0x34ee runnable [0x00007f431df67000]
   java.lang.Thread.State: RUNNABLE
        at java.util.Arrays.copyOf(Arrays.java:2367)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
        at java.lang.StringBuilder.append(StringBuilder.java:132)
        at java.lang.reflect.Method.toGenericString(Method.java:495)
        at java.beans.MethodRef.find(MethodRef.java:79)
        at java.beans.MethodRef.get(MethodRef.java:62)
        at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:207)
        - locked <0x0000000703139d30> (a java.beans.PropertyDescriptor)
        at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2470)
        at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:2250)
        at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3255)
        at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3226)
        - locked <0x00000007defcff68> (a groovy.lang.MetaClassImpl)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:210)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241)
        at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:255)
        at groovy.lang.MetaClassImpl.pickStaticMethod(MetaClassImpl.java:1511)
        at groovy.lang.MetaClassImpl.retrieveStaticMethod(MetaClassImpl.java:1401)
        at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1435)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:875)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:166)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.isCase(ScriptBytecodeAdapter.java:657)
        at org.apache.brooklyn.util.groovy.GroovyJavaMethods.fix(GroovyJavaMethods.java:176)
        at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
        at org.apache.brooklyn.core.effector.MethodEffector.<init>(MethodEffector.java:142)
        at org.apache.brooklyn.core.effector.MethodEffector.<init>(MethodEffector.java:133)
        at org.apache.brooklyn.core.effector.MethodEffector.create(MethodEffector.java:55)
        at org.apache.brooklyn.core.entity.EntityDynamicType.findEffectors(EntityDynamicType.java:317)
        at org.apache.brooklyn.core.entity.EntityDynamicType.<init>(EntityDynamicType.java:82)
        at org.apache.brooklyn.core.entity.EntityDynamicType.<init>(EntityDynamicType.java:76)
        at org.apache.brooklyn.core.objs.BrooklynTypes$ImmutableEntityType.<init>(BrooklynTypes.java:45)
        at org.apache.brooklyn.core.objs.BrooklynTypes$DefinedBrooklynTypeLoader.call(BrooklynTypes.java:88)
        at org.apache.brooklyn.core.objs.BrooklynTypes$DefinedBrooklynTypeLoader.call(BrooklynTypes.java:77)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4739)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
        - locked <0x00000007def8ed90> (a com.google.common.cache.LocalCache$WeakEntry)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4736)
        at org.apache.brooklyn.core.objs.BrooklynTypes.getDefinedBrooklynType(BrooklynTypes.java:113)
        at org.apache.brooklyn.core.objs.BasicSpecParameter$ParseClassParameters.parseParameters(BasicSpecParameter.java:358)
        at org.apache.brooklyn.core.objs.BasicSpecParameter.fromSpec(BasicSpecParameter.java:168)
        at org.apache.brooklyn.core.objs.BasicSpecParameter.addParameters(BasicSpecParameter.java:400)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynEntityDecorationResolver$SpecParameterResolver.decorate(BrooklynEntityDecorationResolver.java:213)

There is a discussion at http://groovy.329449.n5.nabble.com/Deadlock-on-concurrent-metaClass-access-tp3275702p3276031.html (from 2010) about what could cause this.

For us, we should not be calling groovy code here! I think that would get rid of the problem.

The results of top showed that the server was heavily loaded:

top - 11:57:51 up 25 days,  6:05,  2 users,  load average: 5.51, 5.02, 4.74
Tasks: 150 total,   1 running, 149 sleeping,   0 stopped,   0 zombie
Cpu(s): 13.3%us,  1.3%sy,  0.0%ni, 85.2%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   8061144k total,  5431376k used,  2629768k free,   195796k buffers
Swap:  2097148k total,        0k used,  2097148k free,   626448k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
23196 brooklyn  20   0 7673m 3.9g  25m S 243.4 50.3  17960:32 java              
12577 brooklyn  20   0 15028 1124  816 R  3.6  0.0   0:00.40 top                
...

Looking at the 11 "RUNNABLE" threads in jstack (a pooor man's profiling!), only one was in groovy code (shown below):

"brooklyn-jetty-server-8081-qtp97076093-16808" prio=10 tid=0x00007f430c0ff000 nid=0x7ca9 runnable [0x00007f4351cc2000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
        at java.lang.Class.privateGetPublicMethods(Class.java:2743)
        at java.lang.Class.getMethods(Class.java:1480)
        at java.beans.MethodRef.find(MethodRef.java:77)
        at java.beans.MethodRef.get(MethodRef.java:62)
        at java.beans.PropertyDescriptor.getReadMethod(PropertyDescriptor.java:207)
        - locked <0x000000070312ebb8> (a java.beans.PropertyDescriptor)
        at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2470)
        at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:2250)
        at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3255)
        at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3226)
        - locked <0x00000007df006650> (a groovy.lang.MetaClassImpl)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:210)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241)
        at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:251)
        at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:259)
        at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:855)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:888)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:880)
        at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.castToBoolean(DefaultTypeTransformation.java:180)
        at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.booleanUnbox(DefaultTypeTransformation.java:69)
        at org.apache.brooklyn.util.groovy.GroovyJavaMethods.truth(GroovyJavaMethods.java:149)
        at org.apache.brooklyn.util.groovy.GroovyJavaMethods.elvis(GroovyJavaMethods.java:155)
        at org.apache.brooklyn.util.core.flags.FlagUtils.getFlagConfigKeyRecord(FlagUtils.java:277)
        at org.apache.brooklyn.util.core.flags.FlagUtils.findAllFlagsAndConfigKeys(FlagUtils.java:250)
        at org.apache.brooklyn.camp.brooklyn.spi.creation.BrooklynComponentTemplateResolver.findAllFlagsAndConfigKeyValues(BrooklynComponentTemplateResolver.java:376)
        ...

Memory usage was high (according to jmap -histo:live), but not unreasonable given our use of soft references:

 num     #instances         #bytes  class name
----------------------------------------------
   1:      14196525     1728415160  [C
...
Total      73158949     3812605624

Attachments

Issue Links

links to

GitHub Pull Request #513

Activity

People

Assignee:: Unassigned

Reporter:: Aled Sage

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 09/Jan/17 16:40

Updated:: 27/Apr/17 15:55

Resolved:: 27/Apr/17 15:55