[SSHD-1123] ChannelAsyncOutputStream breaks downloads of sftp client by not chunking when the remote window is smaller than the packet size

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.7.0
• Assignee: Lyor Goldstein
• Reporter: Achim Hügen
• Created: 13/Feb/21 16:51
• Updated: 07/Oct/22 00:09
• Resolved: 20/Mar/21 08:11

Description

After the upgrade from 2.4.0 to 2.6.0 we experienced issues with a SFTP Client used in our test automation (Node.JS ssh2). The download of a file hangs after about 1MB of 5MB has been downloaded. 
Although this ssh2 version (0.4.15) is admittedly quite old and meanwhile a fix is available, we are reluctant to rollout 2.6.0 to production where a multitude of sftp clients is used by customers and might experience the same issue. 

Short story: the specific ssh2 client only adjusts its local windows size, if the window size is down to 0. But mina in version 2.6 usually will stop sending data before. So both parties are waiting for each other.

Example: the window size of the client is 1048576 and packet size is 32768.
After successfully requesting 1015808 of file data, when the clients reads the next 32768 bytes, the remote window size as calculated by the MINA is 31706 bytes. Thus the data to send is larger then the remaining space in the window.
The old logic in 2.4 just filled the window to the brim by chunking the data.
Some newly introduced logic (see https://github.com/apache/mina-sshd/commit/536d066349f41a6f6a7f95501506cd1ba66dad7a) now decides to stop sending more data, because the remote window size is smaller then the packet size.
The server waits for the client to send a SSH_MSG_CHANNEL_WINDOW_ADJUST and the client doesn't do this, because the window still has some space.

Unfortunately the commit doesn't contain any reference to why this was changed. As I understand the SSH specification, the packet size is a maximum packet size. So, it should be ok to send a smaller packet.

This is the code causing the issue in ChannelAsyncOutputStream.

if (total > remoteWindowSize) {
    // if we have a big message and there is enough space, send the next chunk
    if (remoteWindowSize >= packetSize) {
        // send the first chunk as we have enough space in the window
        length = packetSize;
    } else {
        // do not chunk when the window is smaller than the packet size
        length = 0;
        // do a defensive copy in case the user reuses the buffer
        IoWriteFutureImpl f = new IoWriteFutureImpl(future.getId(), new ByteArrayBuffer(buffer.getCompactData()));
        f.addListener(w -> future.setValue(w.getException() != null ? w.getException() : w.isWritten()));
        pendingWrite.set(f);
        if (log.isTraceEnabled()) {
            log.trace("doWriteIfPossible({})[resume={}] waiting for window space {}",
                    this, resume, remoteWindowSize);
        }
    }
 

I have attached code for reproducing the issue. To run you need node.js and npm installed:

npm install
nodejs download.js

 

 

Issue Links

• fixes: SSHD-1173 SFTP worker threads got stuck while processing PUT methods against one specific SFTP server
• is related to: SSHD-1207 Large file get using WinSCP with Apache SFTP Server is not working
• is related to: SSHD-1196 sendChunkIfRemoteWindowIsSmallerThanPacketSize flag in ChannelAsyncOutputStream constructor configurable from outside using variable/config file