[IMPALA-5751] impala query KUDU response authentication token signing key expired

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: kudu
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: yyzzjj
• Created: 02/Aug/17 01:09
• Updated: 24/Aug/17 00:27
• Resolved: 04/Aug/17 15:05

Description

Recently moved some tables from kudu 0.8 to kudu 1.3.1 clusters
opt：CREATE TABLE jingo_recommendation.wlogin_monitor_bak ( logintime TIMESTAMP NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, callkey STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, pin STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, ipaddress STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, `timestamp` TIMESTAMP NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, loginresult STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, app_id STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, clientversion STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, osplatform STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, osversion STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, resolution STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, networktype STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, openudid STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, uuid STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, PRIMARY KEY (logintime, callkey, pin, ipaddress) ) PARTITION BY HASH (pin) PARTITIONS 200 STORED AS KUDU TBLPROPERTIES ('numFiles'='0', 'kudu.master_addresses'='172.122.94.12:7051,172.122.94.75:7051,172.122.86.16:7051', 'kudu.table_name'='wlogin_monitor_kudu');

insert into wlogin_monitor_bak select * from wlogin_monitor;
alter table wlogin_monitor rename to del_wlogin_monitor;
alter table wlogin_monitor_bak rename to wlogin_monitor;

before restart impala cluster got next error info when restart impala cluster query could work

client response : ERROR processing query/statement. Error Code: ERROR_STATE, SQL state: Unable to open Kudu table: Runtime error: Client connection negotiation failed: client connection to 172.122.94.75:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token signing key expired
, Query: upsert into community.table_stats values ('jingo_recommendation.add_to_cart_base_agg', '2017-08-01 23:00:23', 1300424919 ).
at com.cloudera.hivecommon.api.HS2Client.executeStatementInternal(Unknown Source)
at com.cloudera.hivecommon.api.HS2Client.executeStatement(Unknown Source)
at com.cloudera.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeHelper(Unknown Source)
at com.cloudera.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.execute(Unknown Source)
at com.cloudera.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.cloudera.jdbc.common.SStatement.execute(Unknown Source)
at jdbc_impala.ImpalaJdbcClient.exec(ImpalaJdbcClient.java:65)

find some logs:
I0801 23:03:02.576370 27639 meta_cache.cc:986] Marking tablet server e652229918674ee7a4fa046e1a93090d (172.22.xx.xx:7050) as failed.
I0801 23:03:02.577934 27886 negotiation.cc:290] Failed RPC negotiation. Trace:
0801 23:03:02.565077 (+ 0us) reactor.cc:373] Submitting negotiation task for client connection to 172.22.86.18:7056
0801 23:03:02.565098 (+ 21us) negotiation.cc:89] Waiting for socket to connect
0801 23:03:02.566074 (+ 976us) client_negotiation.cc:155] Beginning negotiation
0801 23:03:02.566093 (+ 19us) client_negotiation.cc:232] Sending NEGOTIATE NegotiatePB request
0801 23:03:02.568266 (+ 2173us) client_negotiation.cc:247] Received NEGOTIATE NegotiatePB response
0801 23:03:02.568266 (+ 0us) client_negotiation.cc:336] Received NEGOTIATE response from server
0801 23:03:02.568267 (+ 1us) client_negotiation.cc:170] Negotiated authn=TOKEN
0801 23:03:02.568276 (+ 9us) client_negotiation.cc:454] Received TLS_HANDSHAKE response from server
0801 23:03:02.568411 (+ 135us) client_negotiation.cc:442] Sending TLS_HANDSHAKE message to server
0801 23:03:02.568411 (+ 0us) client_negotiation.cc:232] Sending TLS_HANDSHAKE NegotiatePB request
0801 23:03:02.571643 (+ 3232us) client_negotiation.cc:247] Received TLS_HANDSHAKE NegotiatePB response
0801 23:03:02.571643 (+ 0us) client_negotiation.cc:454] Received TLS_HANDSHAKE response from server
0801 23:03:02.572200 (+ 557us) client_negotiation.cc:442] Sending TLS_HANDSHAKE message to server
0801 23:03:02.572201 (+ 1us) client_negotiation.cc:232] Sending TLS_HANDSHAKE NegotiatePB request
0801 23:03:02.575525 (+ 3324us) client_negotiation.cc:247] Received TLS_HANDSHAKE NegotiatePB response
0801 23:03:02.575525 (+ 0us) client_negotiation.cc:454] Received TLS_HANDSHAKE response from server
0801 23:03:02.575618 (+ 93us) client_negotiation.cc:482] Negotiated TLSv1 with cipher suite AES256-SHA
0801 23:03:02.575654 (+ 36us) client_negotiation.cc:232] Sending TOKEN_EXCHANGE NegotiatePB request
0801 23:03:02.577715 (+ 2061us) client_negotiation.cc:258] Received error response from server: Runtime error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token signing key expired
0801 23:03:02.577889 (+ 174us) negotiation.cc:281] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 172.22.86.18:7056: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token signing key expired

Issue Links

• is duplicated by: IMPALA-5829 Insert KUDU Table ERROR: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired