[MESOS-9742] If a HTTP endpoint goes away before finishing sending of data HTTP requests hang

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: libprocess
• Assignee: Unassigned
• Reporter: Benjamin Bannier
• Created: 24/Apr/19 20:00
• Updated: 24/Apr/19 20:01

Description

If a HTTP request is made to a remote that goes away before it finishes sending its data the HTTP request hangs forever.

TEST_P(HTTPTest, NOPE)
{
  Try<inet::Socket > create = inet::Socket::create();
  ASSERT_SOME(create);

  Future<http::Response> response;

  {
    // Post a request which never gets a response.
    Http http;
    EXPECT_CALL(*http.process, body(_));
    response = http::post(http.process->self(), "body/");

    // Wait for some time so the request was posted. There's probably
    // some internal state we could wait for.
    ASSERT_SOME(os::sleep(Milliseconds(300)));
  }

  AWAIT_FAILED(response); // Hangs.
}

While this has likely been an issue for some time it came up with the introduction of agent components which communicate with the agent over HTTP connections, e.g., the for the container daemon or storage local resource providers. Here it becomes hard to reason about the life cycle of async call chains, and also introduces some issues when e.g., executing tests in repetition where we effectively leak sockets (by having Future holding on to the sockets but never reaching a terminal state), see MESOS-8428.

We should evaluate whether we can turn a closed socket into e.g., a failed Future.

Issue Links

• is related to: MESOS-6778 Ability to close connection using http::Response