Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2294

Impalad unable to kinit after several days

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.3.0
    • Fix Version/s: Impala 2.11.0
    • Component/s: Security
    • Labels:
    • Environment:
      impalad version 2.3.0-cdh5-INTERNAL DEBUG (build ca8e31d6d37d92f6b60dd5b657ff0f8cd7c249be)
      Built on Sun, 30 Aug 2015 21:17:36 PST

      Description

      The kerberized stress cluster is unable to execute queries due to failure to kinit. The cluster was setup about 4 days ago and was initially working well. Now the log says

      I0902 04:24:56.235429 23885 authentication.cc:459] Registering impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM, keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
      E0902 04:24:56.236170 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
      /run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM 2>&1', error was: Error(12): Cannot allocate memory
      I0902 05:20:51.236495 23885 authentication.cc:459] Registering impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM, keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
      E0902 05:20:51.237313 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
      /run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM 2>&1', error was: Error(12): Cannot allocate memory
      I0902 06:16:46.237654 23885 authentication.cc:459] Registering impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM, keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
      E0902 06:16:46.238379 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
      /run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM 2>&1', error was: Error(12): Cannot allocate memory
      I0902 07:12:41.238663 23885 authentication.cc:459] Registering impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM, keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
      E0902 07:12:41.239383 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
      /run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM 2>&1', error was: Error(12): Cannot allocate memory
      I0902 08:08:36.239702 23885 authentication.cc:459] Registering impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM, keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
      E0902 08:08:36.240607 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
      /run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/impala-stress-kerberized-5.vpc.cloudera.com@VPC.CLOUDERA.COM 2>&1', error was: Error(12): Cannot alloca
      

      and trying to run a query fails

      [impala-stress-kerberized-5.vpc.cloudera.com:21000] > select count(*) from tpch_10_parquet.lineitem;
      Query: select count(*) from tpch_10_parquet.lineitem
      WARNINGS: 
      Failed to open HDFS file hdfs://impala-stress-kerberized-1.vpc.cloudera.com:8020/user/hive/warehouse/tpch_10_parquet.db/lineitem/e3455d8076e534e6-d52a9c32af878ca9_404434310_data.2.parq
      Error(255): Unknown error 255
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sailesh Sailesh Mukil
                Reporter:
                caseyc casey
              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: