Solr
  1. Solr
  2. SOLR-8189

eTag calculation during http Cache Validation uses unsynchronized WeakHashMap

    Details

      Description

      I found this while looking into a recent jenkins failure where TestDynamicLoading leaked 5 threads: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/14630/

      Stack Trace:
      com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.core.TestDynamicLoading:
         1) Thread[id=11582, name=qtp85907293-11582, state=RUNNABLE, group=TGRP-TestDynamicLoading]
              at java.util.WeakHashMap.get(WeakHashMap.java:403)
              at org.apache.solr.servlet.cache.HttpCacheHeaderUtil.calcEtag(HttpCacheHeaderUtil.java:102)
              at org.apache.solr.servlet.cache.HttpCacheHeaderUtil.doCacheHeaderValidation(HttpCacheHeaderUtil.java:224)
              at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:452)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
              at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
              at org.eclipse.jetty.server.Server.handle(Server.java:499)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
              at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
              at java.lang.Thread.run(Thread.java:745)
         2) Thread[id=11445, name=qtp85907293-11445, state=RUNNABLE, group=TGRP-TestDynamicLoading]
              at java.util.WeakHashMap.get(WeakHashMap.java:403)
              at org.apache.solr.servlet.cache.HttpCacheHeaderUtil.calcEtag(HttpCacheHeaderUtil.java:102)
              at org.apache.solr.servlet.cache.HttpCacheHeaderUtil.doCacheHeaderValidation(HttpCacheHeaderUtil.java:224)
              at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:452)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
              at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
              at org.eclipse.jetty.server.Server.handle(Server.java:499)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
              at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
              at java.lang.Thread.run(Thread.java:745)
         3) Thread[id=11610, name=qtp85907293-11610, state=RUNNABLE, group=TGRP-TestDynamicLoading]
              at java.util.WeakHashMap.get(WeakHashMap.java:403)
              at org.apache.solr.servlet.cache.HttpCacheHeaderUtil.calcEtag(HttpCacheHeaderUtil.java:102)
              at org.apache.solr.servlet.cache.HttpCacheHeaderUtil.doCacheHeaderValidation(HttpCacheHeaderUtil.java:224)
              at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:452)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
              at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
              at org.eclipse.jetty.server.Server.handle(Server.java:499)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
              at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
              at java.lang.Thread.run(Thread.java:745)
         4) Thread[id=11446, name=qtp85907293-11446, state=RUNNABLE, group=TGRP-TestDynamicLoading]
              at java.util.WeakHashMap.get(WeakHashMap.java:403)
              at org.apache.solr.servlet.cache.HttpCacheHeaderUtil.calcEtag(HttpCacheHeaderUtil.java:102)
              at org.apache.solr.servlet.cache.HttpCacheHeaderUtil.doCacheHeaderValidation(HttpCacheHeaderUtil.java:224)
              at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:452)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
              at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
              at org.eclipse.jetty.server.Server.handle(Server.java:499)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
              at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
              at java.lang.Thread.run(Thread.java:745)
         5) Thread[id=11566, name=searcherExecutor-5704-thread-1, state=WAITING, group=TGRP-TestDynamicLoading]
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
              at __randomizedtesting.SeedInfo.seed([AEC7DA9BEB408E47]:0)
      

      4 of these are stuck (in runnable state) at WeakHashMap.get. This is a common problem when WeakHashMap is used without any synchronization and sure enough HttpCacheHeaderUtil uses a WeakHashMap for etagCoreCache without any synchronization.

        Issue Links

          Activity

          Hide
          ASF subversion and git services added a comment -

          Commit 1710218 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1710218 ]

          SOLR-8189: eTag calculation during HTTP Cache Validation uses unsynchronized WeakHashMap causing threads to be stuck in runnable state

          Show
          ASF subversion and git services added a comment - Commit 1710218 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1710218 ] SOLR-8189 : eTag calculation during HTTP Cache Validation uses unsynchronized WeakHashMap causing threads to be stuck in runnable state
          Hide
          ASF subversion and git services added a comment -

          Commit 1710219 from shalin@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1710219 ]

          SOLR-8189: eTag calculation during HTTP Cache Validation uses unsynchronized WeakHashMap causing threads to be stuck in runnable state

          Show
          ASF subversion and git services added a comment - Commit 1710219 from shalin@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1710219 ] SOLR-8189 : eTag calculation during HTTP Cache Validation uses unsynchronized WeakHashMap causing threads to be stuck in runnable state
          Hide
          ASF subversion and git services added a comment -

          Commit 1710240 from shalin@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1710240 ]

          SOLR-8189: Fixed java7 compile issue

          Show
          ASF subversion and git services added a comment - Commit 1710240 from shalin@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1710240 ] SOLR-8189 : Fixed java7 compile issue
          Hide
          Mark Miller added a comment -

          If this dupes SOLR-7423, we should also close that.

          Show
          Mark Miller added a comment - If this dupes SOLR-7423 , we should also close that.
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Mark, I had not seen SOLR-7423. I think Uwe's suggestion is better than my fix. I'll change it to a WeakIdentityMap backed by a ConcurrentHashMap.

          Show
          Shalin Shekhar Mangar added a comment - Thanks Mark, I had not seen SOLR-7423 . I think Uwe's suggestion is better than my fix. I'll change it to a WeakIdentityMap backed by a ConcurrentHashMap.
          Hide
          ASF subversion and git services added a comment -

          Commit 1710366 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1710366 ]

          SOLR-8189: Use WeakIdentityMap.newConcurrentHashMap instead of a synchronized WeakHashMap for better concurrency

          Show
          ASF subversion and git services added a comment - Commit 1710366 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1710366 ] SOLR-8189 : Use WeakIdentityMap.newConcurrentHashMap instead of a synchronized WeakHashMap for better concurrency
          Hide
          ASF subversion and git services added a comment -

          Commit 1710367 from shalin@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1710367 ]

          SOLR-8189: Use WeakIdentityMap.newConcurrentHashMap instead of a synchronized WeakHashMap for better concurrency

          Show
          ASF subversion and git services added a comment - Commit 1710367 from shalin@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1710367 ] SOLR-8189 : Use WeakIdentityMap.newConcurrentHashMap instead of a synchronized WeakHashMap for better concurrency

            People

            • Assignee:
              Shalin Shekhar Mangar
              Reporter:
              Shalin Shekhar Mangar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development