[IMPALA-2598] Server<->Server SSL and Kerberos do not work together

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.5.0, Impala 2.3.2
• Component/s: Security
• Labels: security
• Target Version: Impala 2.5.0, Impala 2.3.2
• Assignee: Sailesh Mukil
• Reporter: Henry Robinson
• Created: 23/Oct/15 23:06
• Updated: 20/Jan/16 18:53
• Resolved: 20/Jan/16 04:17

Description

If server<->server SSL is enabled (with ssl_client_ca_certificate), and Kerberos auth is used between servers, the cluster will fail to start.

The server side hangs in this stack:

Thread 2 (Thread 0x7f778c46b700 (LWP 35069)):
#0  0x000000314a80e54d in read () from /lib64/libpthread.so.0
#1  0x000000314ccb00c1 in ?? () from /usr/lib64/libcrypto.so.10
#2  0x000000314ccae419 in BIO_read () from /usr/lib64/libcrypto.so.10
#3  0x00000039e2625bb8 in ssl3_read_n () from /usr/lib64/libssl.so.10
#4  0x00000039e262627d in ssl3_read_bytes () from /usr/lib64/libssl.so.10
#5  0x00000039e2622700 in ?? () from /usr/lib64/libssl.so.10
#6  0x0000000001ff4320 in apache::thrift::transport::TSSLSocket::read (this=0x30e35c0, buf=0x31be13e "", len=436)
    at src/thrift/transport/TSSLSocket.cpp:167
#7  0x0000000001ff172f in apache::thrift::transport::readAll<apache::thrift::transport::TSocket> (trans=..., 
    buf=0x31be000 "", len=754) at src/thrift/transport/TTransport.h:39
#8  0x0000000000f5dac3 in apache::thrift::transport::TTransport::readAll (this=0x30e35c0, buf=0x31be000 "", len=754)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/thirdparty/thrift-0.9.0/build/include/thrift/transport/TTransport.h:126
#9  0x0000000001187c60 in apache::thrift::transport::TSaslTransport::receiveSaslMessage (this=0x31ac000, 
    status=0x7f778c46a42c, length=0x7f778c46a428)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/be/src/transport/TSaslTransport.cpp:252
#10 0x00000000011870d0 in apache::thrift::transport::TSaslTransport::open (this=0x31ac000)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/be/src/transport/TSaslTransport.cpp:99
#11 0x0000000001183f64 in apache::thrift::transport::TSaslServerTransport::Factory::getTransport (this=0x30db3b0, 
    trans=...) at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/be/src/transport/TSaslServerTransport.cpp:145
#12 0x0000000001ffc928 in apache::thrift::server::TThreadedServer::serve (this=0x30de380)
    at src/thrift/server/TThreadedServer.cpp:162
#13 0x0000000000f6a22e in impala::ThriftServer::ThriftServerEventProcessor::Supervise (this=0x3165d60

and the client in:

#0  0x0000003aa760e54d in read () from /lib64/libpthread.so.0
#1  0x0000003aa9ab00c1 in ?? () from /usr/lib64/libcrypto.so.10
#2  0x0000003aa9aae419 in BIO_read () from /usr/lib64/libcrypto.so.10
#3  0x000000338822ac13 in ssl23_read_bytes () from /usr/lib64/libssl.so.10
#4  0x0000003388229dae in ssl23_connect () from /usr/lib64/libssl.so.10
#5  0x0000000001ffc40c in apache::thrift::transport::TSSLSocket::checkHandshake (this=0x8f4a000)
    at src/thrift/transport/TSSLSocket.cpp:230
#6  0x0000000001ffc5d7 in apache::thrift::transport::TSSLSocket::write (this=0x8f4a000, buf=0x8f9ee00 "\001", len=512)
    at src/thrift/transport/TSSLSocket.cpp:184
#7  0x00000000020008de in write (this=0x94b8e10, 
    buf=0x984f9f0 "\262\370\343\365QG\301P\332\345\006\"\024\213I\210G\036\371\363\236^R\030\220\025+\255\063\023\376\214D\036C\035\257m?\023\222\324\251\207\370\003\324k\376hK\023\002\373\261\206\204EG͘f\202\273\225\377p\326\067U\021\347\273>3\017z\030#1?AĢO\215\321\f\310\003\233\260\205\"\022\266\315\312\333\316\302\033\202l7\005r\310\313\003\350\362\215\rJ(\326#", len=<value optimized out>) at src/thrift/transport/TTransport.h:158
#8  apache::thrift::transport::TBufferedTransport::writeSlow (this=0x94b8e10, 
    buf=0x984f9f0 "\262\370\343\365QG\301P\332\345\006\"\024\213I\210G\036\371\363\236^R\030\220\025+\255\063\023\376\214D\036C\035\257m?\023\222\324\251\207\370\003\324k\376hK\023\002\373\261\206\204EG͘f\202\273\225\377p\326\067U\021\347\273>3\017z\030#1?AĢO\215\321\f\310\003\233\260\205\"\022\266\315\312\333\316\302\033\202l7\005r\310\313\003\350\362\215\rJ(\326#", len=<value optimized out>) at src/thrift/transport/TBufferTransports.cpp:102
#9  0x0000000000f5c045 in apache::thrift::transport::TBufferBase::write (this=0x94b8e10, 
    buf=0x984f800 "`\202\002\356\006\t*\206H\206\367\022\001\002\002\001", len=754)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/thirdparty/thrift-0.9.0/build/include/thrift/transport/TBufferTransports.h:101
#10 0x00000000011962b9 in apache::thrift::transport::TVirtualTransport<apache::thrift::transport::TBufferedTransport, apache::thrift::transport::TBufferBase>::write_virt (this=0x94b8e10, 
    buf=0x984f800 "`\202\002\356\006\t*\206H\206\367\022\001\002\002\001", len=754)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/thirdparty/thrift-0.9.0/build/include/thrift/transport/TVirtualTransport.h:103
#11 0x00000000011a924f in apache::thrift::transport::TTransport::write (this=0x94b8e10, 
    buf=0x984f800 "`\202\002\356\006\t*\206H\206\367\022\001\002\002\001", len=754)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/thirdparty/thrift-0.9.0/build/include/thrift/transport/TTransport.h:158
#12 0x000000000139df68 in apache::thrift::transport::TSaslTransport::sendSaslMessage (this=0x8f97ec0, 
    status=apache::thrift::transport::TSASL_OK, 
    payload=0x984f800 "`\202\002\356\006\t*\206H\206\367\022\001\002\002\001", length=754, flush=true)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/be/src/transport/TSaslTransport.cpp:81
#13 0x0000000001398c49 in apache::thrift::transport::TSaslClientTransport::handleSaslStartMessage (this=0x8f97ec0)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/be/src/transport/TSaslClientTransport.cpp:55
#14 0x000000000139e039 in apache::thrift::transport::TSaslTransport::open (this=0x8f97ec0)
    at /usr/src/debug/impala-2.2.0-cdh5.7.0-SNAPSHOT/be/src/transport/TSaslTransport.cpp:95

Issue Links

• is blocked by: IMPALA-2699 Handle exceptions thrown by TSSLSocket in TSaslTransport.cpp

Sub-Tasks

• Thrift-client cleans openSSL state before using it in the case of the catalog: Resolved