Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-4408

Server hanging on startup

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • 4.1
    • None
    • None
    • None

    • OpenJDK 64-Bit Server VM (23.2-b09 mixed mode)
      Tomcat 7.0
      Eclipse Juno + WTP

    Description

      While starting, the server hangs indefinitely. Everything works fine when I first start the server with no index created yet but if I fill the index then stop and start the server, it hangs. Could it be a lock that is never released?

      Here is what I get in a full thread dump:

      2013-02-06 16:28:52
      Full thread dump OpenJDK 64-Bit Server VM (23.2-b09 mixed mode):

      "searcherExecutor-4-thread-1" prio=10 tid=0x00007fbdfc16a800 nid=0x42c6 in Object.wait() [0x00007fbe0ab10000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00000000c34c1c48> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:503)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1492)
      • locked <0x00000000c34c1c48> (a java.lang.Object)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1312)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1247)
        at org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:94)
        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:213)
        at org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:112)
        at org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:203)
        at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:180)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
        at org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:64)
        at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1594)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

      "coreLoadExecutor-3-thread-1" prio=10 tid=0x00007fbe04194000 nid=0x42c5 in Object.wait() [0x00007fbe0ac11000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00000000c34c1c48> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:503)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1492)
      • locked <0x00000000c34c1c48> (a java.lang.Object)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1312)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1247)
        at org.apache.solr.handler.ReplicationHandler.getIndexVersion(ReplicationHandler.java:495)
        at org.apache.solr.handler.ReplicationHandler.getStatistics(ReplicationHandler.java:518)
        at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo(JmxMonitoredMap.java:232)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:512)
        at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
        at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
        at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:636)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:809)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:607)
        at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:1003)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1033)
        at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
        at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:624)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

      "Thread-3" daemon prio=10 tid=0x00007fbe40742800 nid=0x42c2 waiting on condition [0x00007fbe0ad42000]
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)

      • parking to wait for <0x00000000ef8e6a08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:662)
        at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:340)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:98)
        at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:273)
        at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:254)
        at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:372)
        at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:98)
        at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4461)
      • locked <0x00000000c2e916f8> (a java.util.HashMap)
        at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5133)
        at org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5128)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

      "GC Daemon" daemon prio=10 tid=0x00007fbe405af800 nid=0x42bf in Object.wait() [0x00007fbe383c6000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00000000c2e46c30> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:117)
      • locked <0x00000000c2e46c30> (a sun.misc.GC$LatencyLock)

      "Service Thread" daemon prio=10 tid=0x00007fbe4010c000 nid=0x42bb runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread1" daemon prio=10 tid=0x00007fbe4010a000 nid=0x42ba waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread0" daemon prio=10 tid=0x00007fbe40107000 nid=0x42b9 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "Signal Dispatcher" daemon prio=10 tid=0x00007fbe40104800 nid=0x42b8 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "Finalizer" daemon prio=10 tid=0x00007fbe400ad800 nid=0x42b7 in Object.wait() [0x00007fbe444d0000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00000000c2e46ea8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
      • locked <0x00000000c2e46ea8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

      "Reference Handler" daemon prio=10 tid=0x00007fbe400ab800 nid=0x42b6 in Object.wait() [0x00007fbe445d1000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00000000c2e46c10> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:503)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
      • locked <0x00000000c2e46c10> (a java.lang.ref.Reference$Lock)

      "main" prio=10 tid=0x00007fbe40009800 nid=0x42b1 waiting on condition [0x00007fbe49487000]
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)

      • parking to wait for <0x00000000c34bdc68> (a java.util.concurrent.FutureTask$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.apache.tomcat.util.threads.DedicatedThreadExecutor.execute(DedicatedThreadExecutor.java:56)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5127)
      • locked <0x00000000c2e90d70> (a org.apache.catalina.core.StandardContext)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
      • locked <0x00000000c2e90d70> (a org.apache.catalina.core.StandardContext)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1035)
      • locked <0x00000000c2e90a08> (a org.apache.catalina.core.StandardHost)
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:774)
      • locked <0x00000000c2e90a08> (a org.apache.catalina.core.StandardHost)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
      • locked <0x00000000c2e90a08> (a org.apache.catalina.core.StandardHost)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1035)
      • locked <0x00000000c2e90850> (a org.apache.catalina.core.StandardEngine)
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:291)
      • locked <0x00000000c2e90850> (a org.apache.catalina.core.StandardEngine)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
      • locked <0x00000000c2e90850> (a org.apache.catalina.core.StandardEngine)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:443)
      • locked <0x00000000c2e90850> (a org.apache.catalina.core.StandardEngine)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
      • locked <0x00000000c2e8f640> (a org.apache.catalina.core.StandardService)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:724)
      • locked <0x00000000c2e8f628> (a [Lorg.apache.catalina.Service
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
      • locked <0x00000000c2e471c0> (a org.apache.catalina.core.StandardServer)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:620)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:304)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)

      "VM Thread" prio=10 tid=0x00007fbe400a3000 nid=0x42b5 runnable

      "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fbe40017000 nid=0x42b3 runnable

      "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fbe40019000 nid=0x42b4 runnable

      "VM Periodic Task Thread" prio=10 tid=0x00007fbe40116800 nid=0x42bc waiting on condition

      JNI global references: 189

      Heap
      PSYoungGen total 65280K, used 48438K [0x00000000eba00000, 0x00000000f3980000, 0x0000000100000000)
      eden space 62720K, 73% used [0x00000000eba00000,0x00000000ee6d19b8,0x00000000ef740000)
      from space 2560K, 99% used [0x00000000ef740000,0x00000000ef9bc010,0x00000000ef9c0000)
      to space 2560K, 0% used [0x00000000f3700000,0x00000000f3700000,0x00000000f3980000)
      ParOldGen total 41728K, used 9077K [0x00000000c2e00000, 0x00000000c56c0000, 0x00000000eba00000)
      object space 41728K, 21% used [0x00000000c2e00000,0x00000000c36dd478,0x00000000c56c0000)
      PSPermGen total 25280K, used 25218K [0x00000000b8800000, 0x00000000ba0b0000, 0x00000000c2e00000)
      object space 25280K, 99% used [0x00000000b8800000,0x00000000ba0a0b50,0x00000000ba0b0000)

      Attachments

        1. patch-4408.txt
          2 kB
          Raintung Li

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SOLR-4400 Rapidly opening and closing cores can lead to deadlock

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              fx.bonnet Francois-Xavier Bonnet
              Votes:
              3 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: