[SSHD-754] OOM in sending data for channel

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Workaround
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.0
• Labels: channel stream throttle
• Assignee: Lyor Goldstein
• Reporter: Eugene Petrenko
• Created: 05/Jul/17 12:44
• Updated: 21/Dec/19 08:20
• Resolved: 07/Sep/17 05:55

Description

I have an implementation of SSHD server with the library. It sends gigabytes (e.g. 5GB) of data as command output.

Starting from Putty plink 0.68 (also includes plink 0.69) we started to have OOM errors. Checking memory dumps shown the most of the memory is consumed from the function

org.apache.sshd.common.session.AbstractSession#writePacket(org.apache.sshd.common.util.buffer.Buffer)

In the hprof I see thousands of PendingWriteFuture objects (btw, each holds a reference to a logger instance). And those objects are only created from this function.

It is clear the session is running through rekey. I see the kexState indicating the progress.

Is there a way to artificially limit the sending queue, no matter if related remote window allows sending that enormous amount of data? As of my estimation, the window was reported to be around 1.5 GB or more. Maybe, such huge window size was caused by an arithmetic overflow that is fixed on SSHD-701

Issue Links

• is related to: SSHD-768 Avoid unbounded message queueing when sending large amounts of data to slow clients
• is related to: SSHD-770 Add configurable automatic channel output stream throttling control