[IMPALA-6071] impala-shell hang during Kerberos authentication

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Not Applicable
• Component/s: Clients
• Labels: kerberos supportability
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Matthew Mulder
• Created: 17/Oct/17 23:53
• Updated: 15/Nov/18 00:51
• Resolved: 15/Nov/18 00:51

Description

Four different impala-shell processes hanged in the same place during Kerberos authentication.
Backtrace:

#0  0x00007f2be2ca8928 in poll () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f2be0b378c7 in ?? () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
#2  0x00007f2be0b38104 in ?? () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
#3  0x00007f2be0b3857c in krb5_sendto_kdc () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
#4  0x00007f2be0b0e595 in krb5_tkt_creds_get () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
#5  0x00007f2be0b0e6fd in krb5_get_credentials () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
#6  0x00007f2be0d99c0a in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#7  0x00007f2be0db01e5 in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#8  0x00007f2be0db0831 in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#9  0x00007f2be0da1edd in gss_init_sec_context () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
#10 0x00007f2be0fcedf3 in ?? () from /usr/lib/x86_64-linux-gnu/sasl2/libgssapiv2.so
#11 0x00007f2be16ebc04 in sasl_client_step () from /usr/lib/x86_64-linux-gnu/libsasl2.so.2
#12 0x00007f2be16ec15b in sasl_client_start () from /usr/lib/x86_64-linux-gnu/libsasl2.so.2
#13 0x00007f2be1907ce2 in saslwrapper::ClientImpl::start (this=0x135b178, mechList=..., chosen=..., initialResponse=...) at sasl/saslwrapper.h:301
#14 0x00007f2be190804c in __pyx_f_4sasl_11saslwrapper_6Client_start (__pyx_v_self=0x135b160, __pyx_v_mech_list=0xf96e70, __pyx_skip_dispatch=<optimized out>) at sasl/saslwrapper.cpp:1407
#15 0x00007f2be19085fe in __pyx_pf_4sasl_11saslwrapper_6Client_4start (__pyx_v_mech_list=<optimized out>, __pyx_v_self=<optimized out>) at sasl/saslwrapper.cpp:1483
#16 __pyx_pw_4sasl_11saslwrapper_6Client_5start (__pyx_v_self=<optimized out>, __pyx_v_mech_list=<optimized out>) at sasl/saslwrapper.cpp:1467
#17 0x00000000004f9c46 in PyEval_EvalFrameEx ()
#18 0x00000000005008c2 in PyEval_EvalCodeEx ()
#19 0x00000000004f9ab8 in PyEval_EvalFrameEx ()
#20 0x00000000004f9d02 in PyEval_EvalFrameEx ()
#21 0x00000000004f9d02 in PyEval_EvalFrameEx ()
#22 0x00000000004f9d02 in PyEval_EvalFrameEx ()
#23 0x00000000005008c2 in PyEval_EvalCodeEx ()
#24 0x00000000005012d0 in ?? ()
#25 0x000000000045a8f6 in PyObject_Call ()
#26 0x000000000043f52a in ?? ()
#27 0x000000000045a8f6 in PyObject_Call ()
#28 0x000000000045aba6 in PyEval_CallObjectWithKeywords ()
#29 0x00000000004f1dab in PyInstance_New ()
#30 0x000000000045a8f6 in PyObject_Call ()
#31 0x00000000004f9b6a in PyEval_EvalFrameEx ()
#32 0x00000000004f9d02 in PyEval_EvalFrameEx ()
#33 0x00000000004f9d02 in PyEval_EvalFrameEx ()
#34 0x00000000005008c2 in PyEval_EvalCodeEx ()
#35 0x00000000004990cd in PyRun_FileExFlags ()
#36 0x0000000000499736 in PyRun_SimpleFileExFlags ()
#37 0x000000000049a9cf in Py_Main ()
#38 0x00007f2be2be176d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#39 0x000000000041bb31 in _start ()

The impala-shell processes have the following network connections:

TCP ec2-systest-slave-142f.test:48498->10.17.234.12:21000 (CLOSE_WAIT)
UDP ec2-systest-slave-142f.test:51510->10.17.234.11:kerberos 
TCP ec2-systest-slave-142f.test:40665->10.17.234.11:kerberos (CLOSE_WAIT)

However neither the KDC nor impalad show these connections.
The impala-shell processes are using 100% CPU. These three system calls are being made in a tight loop:

poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}], 2, -106519201) = 1 ([{fd=5, revents=POLLIN}])
read(5, "", 4)                          = 0
gettimeofday({1508277815, 791798}, NULL) = 0

It looks like this is a version 2.9.0 impala-shell running against a version 2.11.0 (pre-release) server.

Issue Links

• relates to: IMPALA-7555 impala-shell can hang in connect in certain cases