[IMPALA-2294] Impalad unable to kinit after several days

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.11.0
• Component/s: Security
• Labels: security stress
• Environment: impalad version 2.3.0-cdh5-INTERNAL DEBUG (build ca8e31d6d37d92f6b60dd5b657ff0f8cd7c249be); Built on Sun, 30 Aug 2015 21:17:36 PST
• Target Version: Product Backlog
• Assignee: Sailesh Mukil
• Reporter: casey
• Created: 03/Sep/15 20:25
• Updated: 23/Nov/17 00:13
• Resolved: 30/Oct/17 16:55

Description

The kerberized stress cluster is unable to execute queries due to failure to kinit. The cluster was setup about 4 days ago and was initially working well. Now the log says

I0902 04:24:56.235429 23885 authentication.cc:459] Registering impala/[e-mail], keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
E0902 04:24:56.236170 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/[e-mail]. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/[e-mail] 2>&1', error was: Error(12): Cannot allocate memory
I0902 05:20:51.236495 23885 authentication.cc:459] Registering impala/[e-mail], keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
E0902 05:20:51.237313 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/[e-mail]. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/[e-mail] 2>&1', error was: Error(12): Cannot allocate memory
I0902 06:16:46.237654 23885 authentication.cc:459] Registering impala/[e-mail], keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
E0902 06:16:46.238379 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/[e-mail]. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/[e-mail] 2>&1', error was: Error(12): Cannot allocate memory
I0902 07:12:41.238663 23885 authentication.cc:459] Registering impala/[e-mail], keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
E0902 07:12:41.239383 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/[e-mail]. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/[e-mail] 2>&1', error was: Error(12): Cannot allocate memory
I0902 08:08:36.239702 23885 authentication.cc:459] Registering impala/[e-mail], keytab file /var/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab
E0902 08:08:36.240607 23885 authentication.cc:471] Failed to obtain Kerberos ticket for principal: impala/[e-mail]. Failed to execute shell cmd: 'kinit -r 1440m -k -t /var
/run/cloudera-scm-agent/process/132-impala-IMPALAD/impala.keytab impala/[e-mail] 2>&1', error was: Error(12): Cannot alloca

and trying to run a query fails

[impala-stress-kerberized-5.vpc.cloudera.com:21000] > select count(*) from tpch_10_parquet.lineitem;
Query: select count(*) from tpch_10_parquet.lineitem
WARNINGS: 
Failed to open HDFS file hdfs://impala-stress-kerberized-1.vpc.cloudera.com:8020/user/hive/warehouse/tpch_10_parquet.db/lineitem/e3455d8076e534e6-d52a9c32af878ca9_404434310_data.2.parq
Error(255): Unknown error 255

Issue Links

• is blocked by: IMPALA-5129 Use Kudu's Kinit code to avoid expensive fork
• Is contained by: IMPALA-5804 Avoid expensive fork for kinit
• is related to: IMPALA-3295 kinit stuck -> impala malfunction
• is related to: IMPALA-975 Improve RunShellProcess() virtual memory behaviour
• relates to: IMPALA-4189 tcmalloc never returns vm space to OS