[MESOS-6799] Scheme/HTTPTest.Endpoints/0 is flaky

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Component/s: libprocess, test
• Labels: flaky flaky-test ssl

Environment

Debian 8, gcc-4.9.2, SSL build w/optimizations and debug symbols

• Assignee: Greg Mann
• Reporter: Benjamin Bannier
• Created: 15/Dec/16 10:17
• Updated: 21/Jan/17 02:19
• Resolved: 21/Jan/17 02:19

Description

Saw Scheme/HTTPTest.Endpoints/0 fail in internal CI with 812e5e3d4e4d9e044a1cfe6cc7eaab10efb499b6,

[03:26:43] :	 [Step 10/11] [ RUN      ] Scheme/HTTPTest.Endpoints/0
[03:26:43]W:	 [Step 10/11] I1215 03:26:43.221824 23530 libevent_ssl_socket.cpp:1141] Socket error: error:00000000:lib(0):func(0):reason(0)
[03:26:43]W:	 [Step 10/11] I1215 03:26:43.448218 23521 openssl.cpp:419] CA file path is unspecified! NOTE: Set CA file path with LIBPROCESS_SSL_CA_FILE=
[03:26:43]W:	 [Step 10/11] I1215 03:26:43.448226 23521 openssl.cpp:424] CA directory path unspecified! NOTE: Set CA directory path with LIBPROCESS_SSL_CA_DIR=
[03:26:43]W:	 [Step 10/11] I1215 03:26:43.448230 23521 openssl.cpp:429] Will not verify peer certificate!
[03:26:43]W:	 [Step 10/11] NOTE: Set LIBPROCESS_SSL_VERIFY_CERT=1 to enable peer certificate verification
[03:26:43]W:	 [Step 10/11] I1215 03:26:43.448231 23521 openssl.cpp:435] Will only verify peer certificate if presented!
[03:26:43]W:	 [Step 10/11] NOTE: Set LIBPROCESS_SSL_REQUIRE_CERT=1 to require peer certificate verification
[03:26:43]W:	 [Step 10/11] I1215 03:26:43.449292 23521 process.cpp:1237] libprocess is initialized on 172.16.10.123:58973 with 8 worker threads
[03:26:43]W:	 [Step 10/11] I1215 03:26:43.452320 23871 process.cpp:3679] Handling HTTP event for process '(75)' with path: '/(75)/body'
[03:26:43]W:	 [Step 10/11] I1215 03:26:43.455099 23870 process.cpp:3679] Handling HTTP event for process '(75)' with path: '/(75)/pipe'
[03:26:43] :	 [Step 10/11] ../../../3rdparty/libprocess/src/tests/http_tests.cpp:275: Failure
[03:26:43] :	 [Step 10/11] (future).failure(): failed to decode body
[03:26:43] :	 [Step 10/11] [  FAILED  ] Scheme/HTTPTest.Endpoints/0, where GetParam() = "https" (234 ms)

I was not able to trigger this failure again in a couple thousand iterations, so there might be some relation to load or other processes running in the system.

We should figure out when this problem first occurred as it might be worthy to backport a fix (if this isn't just a test error).

Issue Links

• is related to: MESOS-6802 SSL socket can lose bytes in the case of EOF