[MESOS-7934] OOM due to LibeventSSLSocket send incorrectly returning 0 after shutdown.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.3, 1.3.2, 1.4.0
• Component/s: libprocess
• Target Version/s: 1.2.3, 1.3.2, 1.4.0
• Epic Link: SSL Tech Debt
• Story Points: 5
• Assignee: Benjamin Mahler
• Reporter: Benjamin Mahler
• Created: 01/Sep/17 23:53
• Updated: 26/Mar/18 20:10
• Resolved: 02/Sep/17 03:32

Description

LibeventSSLSocket can return 0 from send incorrectly, which leads the caller to send the data twice!

See here: https://github.com/apache/mesos/blob/1.3.1/3rdparty/libprocess/src/libevent_ssl_socket.cpp#L396-L398

In some particular cases, it's possible that the caller keeps getting back 0 and loops infinitely, blowing up the memory and OOMing the process.

One example is when a send occurs after a shutdown:

TEST_F(SSLTest, ShutdownThenSend)
{
  Clock::pause();

  Try<Socket> server = setup_server({
      {"LIBPROCESS_SSL_ENABLED", "true"},
      {"LIBPROCESS_SSL_KEY_FILE", key_path().string()},
      {"LIBPROCESS_SSL_CERT_FILE", certificate_path().string()}});

  ASSERT_SOME(server);
  ASSERT_SOME(server.get().address());
  ASSERT_SOME(server.get().address().get().hostname());

  Future<Socket> socket = server.get().accept();

  Clock::settle();
  EXPECT_TRUE(socket.isPending());

  Try<Socket> client = Socket::create(SocketImpl::Kind::SSL);
  ASSERT_SOME(client);
  AWAIT_ASSERT_READY(client->connect(server->address().get()));

  AWAIT_ASSERT_READY(socket);

  EXPECT_SOME(Socket(socket.get()).shutdown());

  // This loops forever!
  AWAIT_FAILED(Socket(socket.get()).send("Hello World"));
}