[MESOS-4069] libevent_ssl_socket assertion fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.24.2, 0.25.1, 0.26.1, 0.27.0
• Component/s: libprocess
• Labels: mesosphere

Environment

ubuntu 14.04

• Target Version/s: 0.27.0
• Epic Link: SSL
• Sprint: Mesosphere Sprint 25
• Story Points: 8
• Assignee: Jojy Varghese
• Reporter: Jojy Varghese
• Shepherd: Joris Van Remoortere
• Created: 04/Dec/15 19:32
• Updated: 26/Nov/18 12:22
• Resolved: 26/Nov/18 12:22

Description

Have been seeing the following socket receive error frequently:

F1204 11:12:47.301839 54104 libevent_ssl_socket.cpp:245] Check failed: length > 0 
*** Check failure stack trace: ***
    @     0x7f73227fe5a6  google::LogMessage::Fail()
    @     0x7f73227fe4f2  google::LogMessage::SendToLog()
    @     0x7f73227fdef4  google::LogMessage::Flush()
    @     0x7f7322800e08  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f73227b93e2  process::network::LibeventSSLSocketImpl::recv_callback()
    @     0x7f73227b9182  process::network::LibeventSSLSocketImpl::recv_callback()
    @     0x7f731cbc75cc  bufferevent_run_deferred_callbacks_locked
    @     0x7f731cbbdc5d  event_base_loop
    @     0x7f73227d9ded  process::EventLoop::run()
    @     0x7f73227a3101  _ZNSt12_Bind_simpleIFPFvvEvEE9_M_invokeIJEEEvSt12_Index_tupleIJXspT_EEE
    @     0x7f73227a305b  std::_Bind_simple<>::operator()()
    @     0x7f73227a2ff4  std::thread::_Impl<>::_M_run()
    @     0x7f731e0d1a40  (unknown)
    @     0x7f731de0a182  start_thread
    @     0x7f731db3730d  (unknown)
    @              (nil)  (unknown)

In this case this was a HTTP get over SSL. The url being:

https://dseasb33srnrn.cloudfront.net:443/registry-v2/docker/registry/v2/blobs/sha256/44/44be94a95984bb47dc3a193f59bf8c04d5e877160b745b119278f38753a6f58f/data?Expires=1449259252&Signature=Q4CQdr1LbxsiYyVebmetrx~lqDgQfHVkGxpbMM3PoISn6r07DXIzBX6~tl1iZx9uXdfr~5awH8Kxwh-y8b0dTV3mLTZAVlneZlHbhBAX9qbYMd180-QvUvrFezwOlSmX4B3idvo-zK0CarUu3Ev1hbJz5y3olwe2ZC~RXHEwzkQ_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q

Steps to reproduce:

1. Run master
2. Run slave from your build directory as as:

 GLOG_v=1;SSL_ENABLED=1;SSL_KEY_FILE=<path_to_key>;SSL_CERT_FILE=<path_to_cert>;sudo -E ./bin/mesos-slave.sh \
      --master=127.0.0.1:5050 \                                                  
      --executor_registration_timeout=5mins \                                    
      --containerizers=mesos  \                                                  
      --isolation=filesystem/linux \                                             
      --image_providers=DOCKER  \                                                
      --docker_puller_timeout=600 \                                              
      --launcher_dir=$MESOS_BUILD_DIR/src/.libs \                                
      --switch_user="false" \                                                    
      --docker_puller="registry"          

3. Run mesos-execute from your build directory as :

                                                        
    ./src/mesos-execute \                                                        
    --master=127.0.0.1:5050 \                                                    
    --command="uname -a"  \                                                      
    --name=test \                                                                
    --docker_image=ubuntu