[SSHD-633] Race condition in command execution and SSH_MSG_CHANNEL_SUCCESS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.0
• Assignee: Lyor Goldstein
• Reporter: Eugene Petrenko
• Created: 27/Jan/16 17:04
• Updated: 01/Feb/16 08:06
• Resolved: 01/Feb/16 08:06

Description

We use the library for production. From time to time we see timeout exceptions from clients that calls our SSH server. It was not clear what is that. Most of the time I saw command suck reading STDIN. We use JSCH as client here. The problem was reproducing rarely.

I connect JSCH 1.51 to SSHD server to execute a command

JSCH client expects to read SSH_MSG_CHANNEL_SUCCESS or SSH_MSG_CHANNEL_FAILURE as response to SSH_MSG_CHANNEL_REQUEST call to execute command.

SSHD implementation calls command's method start and than posts reply with success to the channel. It may easily command thread is able to fill send window with DATA messages. Thus SSH_MSG_CHANNEL_SUCCESS reply is not delivered.

This makes JSCH to wait for the message and fail on timeout. The SSHD server command is simply suck on reading stdin.

Here goes the code I managed to reproduce the issue. What I did is to start sending a huge data block without creating a thread in command's start method. Of course this is a bit incorrect, but it easily make it fail.

The server contains the following command implementation

    final OutputStream out = getOut();
    out.write(new byte[32 * 1024 * 1024]);
    out.flush();

    new Thread(new Runnable() {
      @Override
      public void run() {
        try {
          getIn().read();
          onExit(0);
        } catch (Throwable e) {
          onExit(1, e.getMessage());
        }
      }
    }).start();

Jsch client code is follows

        final JSch j = new JSch();

        final Session session = j.getSession("jonnyzzz", myResource.getHostname(), myResource.getSSHPort());
        session.setTimeout(60_000);
        session.connect();

        final ChannelExec e = (ChannelExec) session.openChannel("exec");
        e.setCommand("test-buffer-underrun");

        final InputStream inputStream = new e.getInputStream();
        e.connect(10_000); //meaningful timeout to reproduce the bug
        ByteStreams.copy(inputStream, FileUtil.nullOutputStream());