Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5751

impala query KUDU response authentication token signing key expired

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Backend
    • Labels:
    • Epic Color:
      ghx-label-9

      Description

      Recently moved some tables from kudu 0.8 to kudu 1.3.1 clusters
      opt:CREATE TABLE jingo_recommendation.wlogin_monitor_bak ( logintime TIMESTAMP NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, callkey STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, pin STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, ipaddress STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, `timestamp` TIMESTAMP NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, loginresult STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, app_id STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, clientversion STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, osplatform STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, osversion STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, resolution STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, networktype STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, openudid STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, uuid STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, PRIMARY KEY (logintime, callkey, pin, ipaddress) ) PARTITION BY HASH (pin) PARTITIONS 200 STORED AS KUDU TBLPROPERTIES ('numFiles'='0', 'kudu.master_addresses'='172.122.94.12:7051,172.122.94.75:7051,172.122.86.16:7051', 'kudu.table_name'='wlogin_monitor_kudu');

      insert into wlogin_monitor_bak select * from wlogin_monitor;
      alter table wlogin_monitor rename to del_wlogin_monitor;
      alter table wlogin_monitor_bak rename to wlogin_monitor;

      before restart impala cluster got next error info when restart impala cluster query could work

      client response : ERROR processing query/statement. Error Code: ERROR_STATE, SQL state: Unable to open Kudu table: Runtime error: Client connection negotiation failed: client connection to 172.122.94.75:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token signing key expired
      , Query: upsert into community.table_stats values ('jingo_recommendation.add_to_cart_base_agg', '2017-08-01 23:00:23', 1300424919 ).
      at com.cloudera.hivecommon.api.HS2Client.executeStatementInternal(Unknown Source)
      at com.cloudera.hivecommon.api.HS2Client.executeStatement(Unknown Source)
      at com.cloudera.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeHelper(Unknown Source)
      at com.cloudera.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.execute(Unknown Source)
      at com.cloudera.jdbc.common.SStatement.executeNoParams(Unknown Source)
      at com.cloudera.jdbc.common.SStatement.execute(Unknown Source)
      at jdbc_impala.ImpalaJdbcClient.exec(ImpalaJdbcClient.java:65)

      find some logs:
      I0801 23:03:02.576370 27639 meta_cache.cc:986] Marking tablet server e652229918674ee7a4fa046e1a93090d (172.22.xx.xx:7050) as failed.
      I0801 23:03:02.577934 27886 negotiation.cc:290] Failed RPC negotiation. Trace:
      0801 23:03:02.565077 (+ 0us) reactor.cc:373] Submitting negotiation task for client connection to 172.22.86.18:7056
      0801 23:03:02.565098 (+ 21us) negotiation.cc:89] Waiting for socket to connect
      0801 23:03:02.566074 (+ 976us) client_negotiation.cc:155] Beginning negotiation
      0801 23:03:02.566093 (+ 19us) client_negotiation.cc:232] Sending NEGOTIATE NegotiatePB request
      0801 23:03:02.568266 (+ 2173us) client_negotiation.cc:247] Received NEGOTIATE NegotiatePB response
      0801 23:03:02.568266 (+ 0us) client_negotiation.cc:336] Received NEGOTIATE response from server
      0801 23:03:02.568267 (+ 1us) client_negotiation.cc:170] Negotiated authn=TOKEN
      0801 23:03:02.568276 (+ 9us) client_negotiation.cc:454] Received TLS_HANDSHAKE response from server
      0801 23:03:02.568411 (+ 135us) client_negotiation.cc:442] Sending TLS_HANDSHAKE message to server
      0801 23:03:02.568411 (+ 0us) client_negotiation.cc:232] Sending TLS_HANDSHAKE NegotiatePB request
      0801 23:03:02.571643 (+ 3232us) client_negotiation.cc:247] Received TLS_HANDSHAKE NegotiatePB response
      0801 23:03:02.571643 (+ 0us) client_negotiation.cc:454] Received TLS_HANDSHAKE response from server
      0801 23:03:02.572200 (+ 557us) client_negotiation.cc:442] Sending TLS_HANDSHAKE message to server
      0801 23:03:02.572201 (+ 1us) client_negotiation.cc:232] Sending TLS_HANDSHAKE NegotiatePB request
      0801 23:03:02.575525 (+ 3324us) client_negotiation.cc:247] Received TLS_HANDSHAKE NegotiatePB response
      0801 23:03:02.575525 (+ 0us) client_negotiation.cc:454] Received TLS_HANDSHAKE response from server
      0801 23:03:02.575618 (+ 93us) client_negotiation.cc:482] Negotiated TLSv1 with cipher suite AES256-SHA
      0801 23:03:02.575654 (+ 36us) client_negotiation.cc:232] Sending TOKEN_EXCHANGE NegotiatePB request
      0801 23:03:02.577715 (+ 2061us) client_negotiation.cc:258] Received error response from server: Runtime error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token signing key expired
      0801 23:03:02.577889 (+ 174us) negotiation.cc:281] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 172.22.86.18:7056: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token signing key expired

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                zzjj yyzzjj
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: