Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6071

impala-shell hang during Kerberos authentication

    XMLWordPrintableJSON

    Details

    • Epic Color:
      ghx-label-5

      Description

      Four different impala-shell processes hanged in the same place during Kerberos authentication.
      Backtrace:

      #0  0x00007f2be2ca8928 in poll () from /lib/x86_64-linux-gnu/libc.so.6
      #1  0x00007f2be0b378c7 in ?? () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
      #2  0x00007f2be0b38104 in ?? () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
      #3  0x00007f2be0b3857c in krb5_sendto_kdc () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
      #4  0x00007f2be0b0e595 in krb5_tkt_creds_get () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
      #5  0x00007f2be0b0e6fd in krb5_get_credentials () from /usr/lib/x86_64-linux-gnu/libkrb5.so.3
      #6  0x00007f2be0d99c0a in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
      #7  0x00007f2be0db01e5 in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
      #8  0x00007f2be0db0831 in ?? () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
      #9  0x00007f2be0da1edd in gss_init_sec_context () from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
      #10 0x00007f2be0fcedf3 in ?? () from /usr/lib/x86_64-linux-gnu/sasl2/libgssapiv2.so
      #11 0x00007f2be16ebc04 in sasl_client_step () from /usr/lib/x86_64-linux-gnu/libsasl2.so.2
      #12 0x00007f2be16ec15b in sasl_client_start () from /usr/lib/x86_64-linux-gnu/libsasl2.so.2
      #13 0x00007f2be1907ce2 in saslwrapper::ClientImpl::start (this=0x135b178, mechList=..., chosen=..., initialResponse=...) at sasl/saslwrapper.h:301
      #14 0x00007f2be190804c in __pyx_f_4sasl_11saslwrapper_6Client_start (__pyx_v_self=0x135b160, __pyx_v_mech_list=0xf96e70, __pyx_skip_dispatch=<optimized out>) at sasl/saslwrapper.cpp:1407
      #15 0x00007f2be19085fe in __pyx_pf_4sasl_11saslwrapper_6Client_4start (__pyx_v_mech_list=<optimized out>, __pyx_v_self=<optimized out>) at sasl/saslwrapper.cpp:1483
      #16 __pyx_pw_4sasl_11saslwrapper_6Client_5start (__pyx_v_self=<optimized out>, __pyx_v_mech_list=<optimized out>) at sasl/saslwrapper.cpp:1467
      #17 0x00000000004f9c46 in PyEval_EvalFrameEx ()
      #18 0x00000000005008c2 in PyEval_EvalCodeEx ()
      #19 0x00000000004f9ab8 in PyEval_EvalFrameEx ()
      #20 0x00000000004f9d02 in PyEval_EvalFrameEx ()
      #21 0x00000000004f9d02 in PyEval_EvalFrameEx ()
      #22 0x00000000004f9d02 in PyEval_EvalFrameEx ()
      #23 0x00000000005008c2 in PyEval_EvalCodeEx ()
      #24 0x00000000005012d0 in ?? ()
      #25 0x000000000045a8f6 in PyObject_Call ()
      #26 0x000000000043f52a in ?? ()
      #27 0x000000000045a8f6 in PyObject_Call ()
      #28 0x000000000045aba6 in PyEval_CallObjectWithKeywords ()
      #29 0x00000000004f1dab in PyInstance_New ()
      #30 0x000000000045a8f6 in PyObject_Call ()
      #31 0x00000000004f9b6a in PyEval_EvalFrameEx ()
      #32 0x00000000004f9d02 in PyEval_EvalFrameEx ()
      #33 0x00000000004f9d02 in PyEval_EvalFrameEx ()
      #34 0x00000000005008c2 in PyEval_EvalCodeEx ()
      #35 0x00000000004990cd in PyRun_FileExFlags ()
      #36 0x0000000000499736 in PyRun_SimpleFileExFlags ()
      #37 0x000000000049a9cf in Py_Main ()
      #38 0x00007f2be2be176d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
      #39 0x000000000041bb31 in _start ()

      The impala-shell processes have the following network connections:

      TCP ec2-systest-slave-142f.test:48498->10.17.234.12:21000 (CLOSE_WAIT)
      UDP ec2-systest-slave-142f.test:51510->10.17.234.11:kerberos 
      TCP ec2-systest-slave-142f.test:40665->10.17.234.11:kerberos (CLOSE_WAIT)

      However neither the KDC nor impalad show these connections.
      The impala-shell processes are using 100% CPU. These three system calls are being made in a tight loop:

      poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}], 2, -106519201) = 1 ([{fd=5, revents=POLLIN}])
      read(5, "", 4)                          = 0
      gettimeofday({1508277815, 791798}, NULL) = 0

      It looks like this is a version 2.9.0 impala-shell running against a version 2.11.0 (pre-release) server.

        Attachments

        1. krb5kdc.log.bz2
          5 kB
          Matthew Mulder

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mmulder Matthew Mulder
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: