Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6726

Catalog server's kerberos ticket gets deleted after 'ticket_lifetime' on SLES11

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.11.0, Impala 2.12.0
    • Fix Version/s: Impala 2.12.0
    • Component/s: Security
    • Labels:
    • Docs Text:
      Hide
      Workaround for this issue:

      On 2.11.0, set --use_kudu_kinit=false in Impala startup flag.
      On 2.12.0, set --use_kudu_kinit=false and --use_krpc=false in Impala startup flags.
      Show
      Workaround for this issue: On 2.11.0, set --use_kudu_kinit=false in Impala startup flag. On 2.12.0, set --use_kudu_kinit=false and --use_krpc=false in Impala startup flags.
    • Target Version:
    • Epic Color:
      ghx-label-9

      Description

      On SLES11, it was noticed that after 'ticket_lifetime', the kerberos ticket gets deleted by the Java krb5 library. Michael Brown noticed this from 2.11, and we confirmed that it shows up in 2.12 as well.

      I turned on the Java kerberos debug logging and found this in the log messages:

      W0322 07:51:43.617998 12118 UserGroupInformation.java:1403] Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1521730246019
      >>>DEBUG <CCacheInputStream>  client principal is impala/mikeb-sles11-1.vpc.cloudera.com@VPC.CLOUDERA.COM
      >>>DEBUG <CCacheInputStream> server principal is krbtgt/VPC.CLOUDERA.COM@VPC.CLOUDERA.COM
      >>>DEBUG <CCacheInputStream> key type: 16
      >>>DEBUG <CCacheInputStream> auth time: Thu Mar 22 07:21:58 PDT 2018
      >>>DEBUG <CCacheInputStream> start time: Thu Mar 22 07:51:46 PDT 2018
      >>>DEBUG <CCacheInputStream> end time: Thu Mar 22 07:51:58 PDT 2018
      >>>DEBUG <CCacheInputStream> renew_till time: Thu Mar 22 07:51:58 PDT 2018
      >>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL;
      Found ticket for impala/mikeb-sles11-1.vpc.cloudera.com@VPC.CLOUDERA.COM to go to krbtgt/VPC.CLOUDERA.COM@VPC.CLOUDERA.COM expiring on Thu Mar 22 07:51:58 PDT 2018
      Removed and destroyed the expired Ticket
      Destroyed KerberosTicket
      W0322 07:52:04.195199 12201 UserGroupInformation.java:1920] PriviledgedActionException as:impala/mikeb-sles11-1.vpc.cloudera.com@VPC.CLOUDERA.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      W0322 07:52:04.200016 12201 UserGroupInformation.java:1403] Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1521730306038
      

      The backend ticket acquisition thread however keeps running and claiming to have re-acquired a ticket every 'ticket_lifetime' period.

      I tried turning off the 'use_kudu_kinit' flag and this bug didn't show up in that mode.

      Still investigating the bug.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kwho Michael Ho
                Reporter:
                sailesh Sailesh Mukil
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: