[JCLOUDS-1607] Deadlock on SimpleDateFormatDateService.rfc822SimpleDateFormat monitor

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Component/s: jclouds-blobstore
• Assignee: Unassigned
• Reporter: John Calcote
• Created: 04/May/22 18:00
• Updated: 05/May/22 15:06

Description

I have a deadlock scenario where jstack shows 14 threads BLOCKED on the object monitor, and one other thread as the owner. The owning thread is not anywhere near that code - it's, in fact, setting in the thread pool waiting for work to do (owner and one blocked thread shown below for brevity):

"clrd-normalpri-13" Id=165 BLOCKED on java.text.SimpleDateFormat@1b9ff8be owned by "clrd-normalpri-12" Id=164
        at org.jclouds.date.internal.SimpleDateFormatDateService.rfc822DateParse(SimpleDateFormatDateService.java:103)
        -  blocked on java.text.SimpleDateFormat@1b9ff8be
        at org.jclouds.blobstore.functions.ParseSystemAndUserMetadataFromHeaders.parseLastModifiedOrThrowException(ParseSystemAndUserMetadataFromHeaders.java:99)
        at org.jclouds.blobstore.functions.ParseSystemAndUserMetadataFromHeaders.apply(ParseSystemAndUserMetadataFromHeaders.java:72)
        at org.jclouds.s3.functions.ParseObjectMetadataFromHeaders.apply(ParseObjectMetadataFromHeaders.java:62)
        at org.jclouds.s3.functions.ParseObjectFromHeadersAndHttpContent.apply(ParseObjectFromHeadersAndHttpContent.java:48)
        at org.jclouds.s3.functions.ParseObjectFromHeadersAndHttpContent.apply(ParseObjectFromHeadersAndHttpContent.java:34)
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
        ...

        Number of locked synchronizers = 1
        - java.util.concurrent.ThreadPoolExecutor$Worker@39133244

"clrd-normalpri-12" Id=164 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f32283d
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f32283d
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

This issue is not easily reproduced, and frankly I can't understand how it can happen. The JVM is supposed to guarantee it doesn't, so I have no suggestions as to how one might fix it. SO question posted at: https://stackoverflow.com/questions/72117726/jclouds-multiple-threads-blocked-on-object-monitor-owned-by-thread-waiting-in-p - maybe someone out there has a clue.