Description
Hey guys,
Recently I've encountered one remote SFTP server, that causes SFTP worker threads to enter TIMED_WAITING state, from which they do not recover. Please, take a look into this thread dump:
SFTP-worker-3 #2155 prio=5 os_prio=0 cpu=61692.09ms elapsed=1136151.86s tid=0x00007fe41c005800 nid=0x18808 in Object.wait() [0x00007fe145b1d000]SFTP-worker-3" #2155 prio=5 os_prio=0 cpu=61692.09ms elapsed=1136151.86s tid=0x00007fe41c005800 nid=0x18808 in Object.wait() [0x00007fe145b1d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.10/Native Method) - waiting on <no object reference available> at org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69) - waiting to re-lock in wait() <0x00000006e3bbc420> (a org.apache.sshd.common.channel.IoWriteFutureImpl) at org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:109) at org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:39) at org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:30) at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43) at org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292) at org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.flush(SftpOutputStreamAsync.java:210) at org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.write(SftpOutputStreamAsync.java:141) at java.io.InputStream.transferTo(java.base@11.0.10/InputStream.java:705) at com.mina.command.Put.replyInto(Put.java:55) at com.sftpserver.MinaSftpHandler.channelReadInternal(MinaSftpHandler.java:251) at com.sftpserver.MinaSftpHandler.lambda$channelRead$0(MinaSftpHandler.java:125) at com.sftpserver.MinaSftpHandler$$Lambda$392/0x0000000800764040.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.10/Executors.java:515) at java.util.concurrent.FutureTask.run(java.base@11.0.10/FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.10/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.10/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)
Seems like the thread is waiting for lock in org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java), I'm currently not sure what thread holds the lock and why it's not ever released.
Also, I'm not able to reproduce this, but it constantly happens to all the PUT methods targeting one specific remote server (which I don't own), so I suppose there would be something odd with the remote server, but it doesn't mean, that we shouldn't be able to deal with that.
Have you ever seen such behavior? Could it be Mina sshd's bug? (seems like verify() in org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292) is called without any timeout, which then defaults to LONG.MAX here: at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43), shouldn't we have configurable timeout here? or what's the point of default timeout? what are we really waiting for in at org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69)?)
Attachments
Issue Links
- is fixed by
-
SSHD-1123 ChannelAsyncOutputStream breaks downloads of sftp client by not chunking when the remote window is smaller than the packet size
- Resolved