Uploaded image for project: 'Commons VFS'
  1. Commons VFS
  2. VFS-343

Deadlock occurs in HTTP or WebDAV when all the connections are in use.

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0
    • None

    Description

      The stacktrace below shows several threads all waiting on a FileSystem lock. The thread holding that lock is waiting for an HTTP connection from the connection pool. However, each of the threads waiting on the FileSystem lock has already acquired a connection from the pool. This issue can be resolved by reducing the amount of locking that occurs on the FileSystem objects. In this case the threads are attempting to get the lock to increment a counter. Changing the counters to AtomicIntegers will resolve the problem.

      Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode):

      "Attach Listener" daemon prio=10 tid=0x0000000053834400 nid=0x23b1 waiting on condition [0x0000000000000000..0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0x00000000538ff400 nid=0x21f0 in Object.wait() [0x000000004077e000..0x000000004077ec10]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002aab82694428> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
      • locked <0x00002aab82694428> (a java.lang.ref.ReferenceQueue$Lock)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)

      "Thread 2" prio=10 tid=0x00002aabc00a0800 nid=0x21ef waiting for monitor entry [0x0000000040d17000..0x0000000040d17c90]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.commons.vfs.provider.AbstractFileSystem.streamOpened(AbstractFileSystem.java:602)

      • waiting to lock <0x00002aab827f92c8> (a org.apache.commons.vfs.provider.webdav.WebdavFileSystem)
        at org.apache.commons.vfs.provider.DefaultFileContent.streamOpened(DefaultFileContent.java:93)
        at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:373)
        at org.apache.commons.configuration.VFSFileSystem.getInputStream(VFSFileSystem.java:111)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:311)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:251)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229)
        at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:180)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClientProduct(ConfigurationSynchronizer.java:862)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClient(ConfigurationSynchronizer.java:829)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.run(ConfigurationSynchronizer.java:416)
        at java.lang.Thread.run(Thread.java:619)

      "Thread 3" prio=10 tid=0x00002aabc009f800 nid=0x21ec in Object.wait() [0x0000000042593000..0x0000000042593d10]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002aab826c67a8> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:509)
      • locked <0x00002aab826c67a8> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:394)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:152)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
        at org.apache.commons.vfs.provider.webdav.WebdavFileObject.execute(WebdavFileObject.java:349)
        at org.apache.commons.vfs.provider.webdav.WebdavFileObject.getProperties(WebdavFileObject.java:442)
        at org.apache.commons.vfs.provider.webdav.WebdavFileObject.getProperties(WebdavFileObject.java:417)
        at org.apache.commons.vfs.provider.webdav.WebdavFileObject.getProperty(WebdavFileObject.java:410)
        at org.apache.commons.vfs.provider.webdav.WebdavFileObject.getProperty(WebdavFileObject.java:402)
        at org.apache.commons.vfs.provider.webdav.WebdavFileObject.isDirectory(WebdavFileObject.java:382)
        at org.apache.commons.vfs.provider.webdav.WebdavFileObject.doGetType(WebdavFileObject.java:112)
        at org.apache.commons.vfs.provider.AbstractFileObject.getType(AbstractFileObject.java:441)
      • locked <0x00002aab827f92c8> (a org.apache.commons.vfs.provider.webdav.WebdavFileSystem)
        at org.apache.commons.vfs.provider.AbstractFileObject.exists(AbstractFileObject.java:424)
        at org.apache.commons.configuration.VFSFileSystem.locateFromURL(VFSFileSystem.java:293)
        at org.apache.commons.configuration.ConfigurationUtils.locate(ConfigurationUtils.java:466)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:245)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229)
        at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:180)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClientProduct(ConfigurationSynchronizer.java:862)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClient(ConfigurationSynchronizer.java:829)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.run(ConfigurationSynchronizer.java:416)
        at java.lang.Thread.run(Thread.java:619)

      "Thread 4" prio=10 tid=0x00002aabc00ce000 nid=0x21eb waiting for monitor entry [0x0000000042492000..0x0000000042492d90]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.commons.vfs.provider.AbstractFileSystem.streamOpened(AbstractFileSystem.java:602)

      • waiting to lock <0x00002aab827f92c8> (a org.apache.commons.vfs.provider.webdav.WebdavFileSystem)
        at org.apache.commons.vfs.provider.DefaultFileContent.streamOpened(DefaultFileContent.java:93)
        at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:373)
        at org.apache.commons.configuration.VFSFileSystem.getInputStream(VFSFileSystem.java:111)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:311)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:251)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229)
        at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:180)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClientProduct(ConfigurationSynchronizer.java:862)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClient(ConfigurationSynchronizer.java:829)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.run(ConfigurationSynchronizer.java:416)
        at java.lang.Thread.run(Thread.java:619)

      "Thread 5" prio=10 tid=0x00002aabc00d2000 nid=0x21ea waiting for monitor entry [0x0000000042391000..0x0000000042391a10]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.commons.vfs.provider.AbstractFileSystem.streamOpened(AbstractFileSystem.java:602)

      • waiting to lock <0x00002aab827f92c8> (a org.apache.commons.vfs.provider.webdav.WebdavFileSystem)
        at org.apache.commons.vfs.provider.DefaultFileContent.streamOpened(DefaultFileContent.java:93)
        at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:373)
        at org.apache.commons.configuration.VFSFileSystem.getInputStream(VFSFileSystem.java:111)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:311)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:251)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229)
        at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:180)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClientProduct(ConfigurationSynchronizer.java:862)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClient(ConfigurationSynchronizer.java:829)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.run(ConfigurationSynchronizer.java:416)
        at java.lang.Thread.run(Thread.java:619)

      "Thread 6" prio=10 tid=0x00002aabc00af400 nid=0x21e9 waiting for monitor entry [0x0000000042290000..0x0000000042290a90]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.commons.vfs.provider.AbstractFileSystem.resolveFile(AbstractFileSystem.java:294)

      • waiting to lock <0x00002aab827f92c8> (a org.apache.commons.vfs.provider.webdav.WebdavFileSystem)
        at org.apache.commons.vfs.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:86)
        at org.apache.commons.vfs.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:66)
        at org.apache.commons.vfs.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:692)
        at org.apache.commons.vfs.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:648)
        at org.apache.commons.vfs.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:604)
        at org.apache.commons.configuration.VFSFileSystem.getInputStream(VFSFileSystem.java:99)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:311)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:251)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229)
        at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:180)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClientProduct(ConfigurationSynchronizer.java:862)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClient(ConfigurationSynchronizer.java:829)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.run(ConfigurationSynchronizer.java:416)
        at java.lang.Thread.run(Thread.java:619)

      "Thread 7" prio=10 tid=0x00002aabc001cc00 nid=0x21e8 waiting for monitor entry [0x000000004218f000..0x000000004218fb10]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.commons.vfs.provider.AbstractFileSystem.streamOpened(AbstractFileSystem.java:602)

      • waiting to lock <0x00002aab827f92c8> (a org.apache.commons.vfs.provider.webdav.WebdavFileSystem)
        at org.apache.commons.vfs.provider.DefaultFileContent.streamOpened(DefaultFileContent.java:93)
        at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:373)
        at org.apache.commons.configuration.VFSFileSystem.getInputStream(VFSFileSystem.java:111)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:311)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:251)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229)
        at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:180)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClientProduct(ConfigurationSynchronizer.java:862)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClient(ConfigurationSynchronizer.java:829)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.run(ConfigurationSynchronizer.java:416)
        at java.lang.Thread.run(Thread.java:619)

      "org.apache.commons.vfs.cache.SoftRefFilesCache$SoftRefReleaseThread" daemon prio=10 tid=0x00002aabc001ec00 nid=0x21e7 in Object.wait() [0x000000004208e000..0x000000004208eb90]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002aaabe1f0198> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
      • locked <0x00002aaabe1f0198> (a java.lang.ref.ReferenceQueue$Lock)
        at org.apache.commons.vfs.cache.SoftRefFilesCache$SoftRefReleaseThread.run(SoftRefFilesCache.java:79)

      "Low Memory Detector" daemon prio=10 tid=0x000000005360bc00 nid=0x21e4 runnable [0x0000000000000000..0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "CompilerThread1" daemon prio=10 tid=0x0000000053609000 nid=0x21e3 waiting on condition [0x0000000000000000..0x0000000041d8a4c0]
      java.lang.Thread.State: RUNNABLE

      "CompilerThread0" daemon prio=10 tid=0x0000000053605400 nid=0x21e2 waiting on condition [0x0000000000000000..0x000000004165c5d0]
      java.lang.Thread.State: RUNNABLE

      "Signal Dispatcher" daemon prio=10 tid=0x0000000053603400 nid=0x21e1 runnable [0x0000000000000000..0x0000000040ed66a0]
      java.lang.Thread.State: RUNNABLE

      "Finalizer" daemon prio=10 tid=0x00000000535e0c00 nid=0x21e0 in Object.wait() [0x000000004155c000..0x000000004155ca90]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002aaabe1f0278> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
      • locked <0x00002aaabe1f0278> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

      "Reference Handler" daemon prio=10 tid=0x00000000535df000 nid=0x21df in Object.wait() [0x000000004145b000..0x000000004145bb10]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x00002aaabe1f2370> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      • locked <0x00002aaabe1f2370> (a java.lang.ref.Reference$Lock)

      "Thread 1" prio=10 tid=0x0000000053427400 nid=0x21d5 waiting for monitor entry [0x00000000412af000..0x00000000412afed0]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.commons.vfs.provider.AbstractFileSystem.streamOpened(AbstractFileSystem.java:602)

      • waiting to lock <0x00002aab827f92c8> (a org.apache.commons.vfs.provider.webdav.WebdavFileSystem)
        at org.apache.commons.vfs.provider.DefaultFileContent.streamOpened(DefaultFileContent.java:93)
        at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:373)
        at org.apache.commons.configuration.VFSFileSystem.getInputStream(VFSFileSystem.java:111)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:311)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:251)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229)
        at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:180)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClientProduct(ConfigurationSynchronizer.java:862)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.synchronizeClient(ConfigurationSynchronizer.java:829)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.run(ConfigurationSynchronizer.java:416)
        at com.diginsite.services.configsync.ConfigurationSynchronizer.main(ConfigurationSynchronizer.java:1790)

      "VM Thread" prio=10 tid=0x00000000535d9800 nid=0x21de runnable

      "GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000053432000 nid=0x21d6 runnable

      "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000053433800 nid=0x21d7 runnable

      "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000053435400 nid=0x21d8 runnable

      "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000053436c00 nid=0x21d9 runnable

      "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000053438400 nid=0x21da runnable

      "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005343a000 nid=0x21db runnable

      "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005343b800 nid=0x21dc runnable

      "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005343d000 nid=0x21dd runnable

      "VM Periodic Task Thread" prio=10 tid=0x000000005360e000 nid=0x21e5 waiting on condition

      JNI global references: 902

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            rgoers Ralph Goers
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment