I0619 09:54:40.593186 30577 impala-server.cc:1060] 2b4de6f8564f0cef:9ff7b87500000000] Registered query query_id=2b4de6f8564f0cef:9ff7b87500000000 session_id=dd4ac6c213c15a51:54959ca4493ee095 I0619 09:54:40.629745 30577 Frontend.java:1245] 2b4de6f8564f0cef:9ff7b87500000000] Analyzing query: select credit_card_number from test_sensitive_data_redaction_table_bnomt where credit_card_number = 'XXXX-XXXX-XXXX-XXXX' order by credit_card_number I0619 09:54:40.744582 30577 FeSupport.java:285] 2b4de6f8564f0cef:9ff7b87500000000] Requesting prioritized load of table(s): default.test_sensitive_data_redaction_table_bnomt I0619 09:54:40.756475 30577 authentication.cc:173] 2b4de6f8564f0cef:9ff7b87500000000] SASL message (Kerberos (internal)): GSSAPI client step 1 I0619 09:54:40.762725 30577 authentication.cc:173] 2b4de6f8564f0cef:9ff7b87500000000] SASL message (Kerberos (internal)): GSSAPI client step 1 I0619 09:54:40.762979 30577 authentication.cc:173] 2b4de6f8564f0cef:9ff7b87500000000] SASL message (Kerberos (internal)): GSSAPI client step 2 I0619 09:54:44.401727 30577 Frontend.java:1285] 2b4de6f8564f0cef:9ff7b87500000000] Analysis finished. I0619 09:54:44.678135 30596 admission-controller.cc:627] 2b4de6f8564f0cef:9ff7b87500000000] Schedule for id=2b4de6f8564f0cef:9ff7b87500000000 in pool_name=root.systest per_host_mem_estimate=44.02 MB PoolConfig: max_requests=-1 max_queued=200 max_mem=-1.00 B I0619 09:54:44.678220 30596 admission-controller.cc:632] 2b4de6f8564f0cef:9ff7b87500000000] Stats: agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0, local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0, backend_mem_reserved=0) I0619 09:54:44.678239 30596 admission-controller.cc:664] 2b4de6f8564f0cef:9ff7b87500000000] Admitted query id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:54:44.678274 30596 coordinator.cc:95] 2b4de6f8564f0cef:9ff7b87500000000] Exec() query_id=2b4de6f8564f0cef:9ff7b87500000000 stmt=select credit_card_number from test_sensitive_data_redaction_table_bnomt where credit_card_number = 'XXXX-XXXX-XXXX-XXXX' order by credit_card_number I0619 09:54:44.678397 30596 coordinator.cc:361] 2b4de6f8564f0cef:9ff7b87500000000] starting execution on 2 backends for query_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:54:44.758570 30598 impala-internal-service.cc:50] 2b4de6f8564f0cef:9ff7b87500000000] ExecQueryFInstances(): query_id=2b4de6f8564f0cef:9ff7b87500000000 coord=quasar-epvzqp-4.vpc.cloudera.com:22000 #instances=1 I0619 09:54:44.758827 30600 query-state.cc:624] 2b4de6f8564f0cef:9ff7b87500000000] Executing instance. instance_id=2b4de6f8564f0cef:9ff7b87500000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=1 #in-flight=1 I0619 09:54:44.759552 30596 coordinator.cc:375] 2b4de6f8564f0cef:9ff7b87500000000] started execution on 2 backends for query_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:54:49.763633 30599 kudu-util.h:108] 2b4de6f8564f0cef:9ff7b87500000000] ReportExecStatus() RPC failed: Not authorized: Client connection negotiation failed: client connection to 10.65.7.41:27000: FATAL_UNAUTHORIZED: Not authorized: expected TLS_HANDSHAKE step: SASL_INITIATE W0619 09:54:49.763656 30599 query-state.cc:396] 2b4de6f8564f0cef:9ff7b87500000000] Failed to send ReportExecStatus() RPC for query 2b4de6f8564f0cef:9ff7b87500000000. Consecutive failed reports = 1 I0619 09:54:59.767486 30599 kudu-util.h:108] 2b4de6f8564f0cef:9ff7b87500000000] ReportExecStatus() RPC failed: Not authorized: Client connection negotiation failed: client connection to 10.65.7.41:27000: FATAL_UNAUTHORIZED: Not authorized: expected TLS_HANDSHAKE step: SASL_INITIATE W0619 09:54:59.767529 30599 query-state.cc:396] 2b4de6f8564f0cef:9ff7b87500000000] Failed to send ReportExecStatus() RPC for query 2b4de6f8564f0cef:9ff7b87500000000. Consecutive failed reports = 2 I0619 09:55:14.771435 30599 kudu-util.h:108] 2b4de6f8564f0cef:9ff7b87500000000] ReportExecStatus() RPC failed: Not authorized: Client connection negotiation failed: client connection to 10.65.7.41:27000: FATAL_UNAUTHORIZED: Not authorized: expected TLS_HANDSHAKE step: SASL_INITIATE W0619 09:55:14.773367 30599 query-state.cc:396] 2b4de6f8564f0cef:9ff7b87500000000] Failed to send ReportExecStatus() RPC for query 2b4de6f8564f0cef:9ff7b87500000000. Consecutive failed reports = 3 E0619 09:55:14.773388 30599 query-state.cc:424] 2b4de6f8564f0cef:9ff7b87500000000] Cancelling fragment instances due to failure to reach the coordinator. (ReportExecStatus() RPC failed: Not authorized: Client connection negotiation failed: client connection to 10.65.7.41:27000: FATAL_UNAUTHORIZED: Not authorized: expected TLS_HANDSHAKE step: SASL_INITIATE ). Query 2b4de6f8564f0cef:9ff7b87500000000 may hang. See IMPALA-2990. I0619 09:55:14.773414 30599 query-state.cc:649] 2b4de6f8564f0cef:9ff7b87500000000] Cancel: query_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:55:14.773424 30599 krpc-data-stream-mgr.cc:326] 2b4de6f8564f0cef:9ff7b87500000000] cancelling all streams for fragment_instance_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:55:14.773528 30600 krpc-data-stream-mgr.cc:295] 2b4de6f8564f0cef:9ff7b87500000000] DeregisterRecvr(): fragment_instance_id=2b4de6f8564f0cef:9ff7b87500000000, node=2 I0619 09:55:14.773537 30597 coordinator.cc:529] ExecState: query id=2b4de6f8564f0cef:9ff7b87500000000 finstance=2b4de6f8564f0cef:9ff7b87500000000 on host=quasar-epvzqp-4.vpc.cloudera.com (EXECUTING -> ERROR) status=Cancelled I0619 09:55:14.773566 30597 coordinator-backend-state.cc:445] Sending CancelQueryFInstances rpc for query_id=2b4de6f8564f0cef:9ff7b87500000000 backend=10.65.0.101:27000 I0619 09:55:14.773797 30599 query-state.cc:272] 2b4de6f8564f0cef:9ff7b87500000000] UpdateBackendExecState(): last report for 2b4de6f8564f0cef:9ff7b87500000000 I0619 09:55:14.773793 30600 query-state.cc:632] 2b4de6f8564f0cef:9ff7b87500000000] Instance completed. instance_id=2b4de6f8564f0cef:9ff7b87500000000 #in-flight=0 status=CANCELLED: Cancelled I0619 09:55:14.777575 30599 kudu-util.h:108] 2b4de6f8564f0cef:9ff7b87500000000] ReportExecStatus() RPC failed: Not authorized: Client connection negotiation failed: client connection to 10.65.7.41:27000: FATAL_UNAUTHORIZED: Not authorized: expected TLS_HANDSHAKE step: SASL_INITIATE W0619 09:55:14.777604 30599 query-state.cc:396] 2b4de6f8564f0cef:9ff7b87500000000] Failed to send ReportExecStatus() RPC for query 2b4de6f8564f0cef:9ff7b87500000000. Consecutive failed reports = 4 E0619 09:55:14.777621 30599 query-state.cc:424] 2b4de6f8564f0cef:9ff7b87500000000] Cancelling fragment instances due to failure to reach the coordinator. (ReportExecStatus() RPC failed: Not authorized: Client connection negotiation failed: client connection to 10.65.7.41:27000: FATAL_UNAUTHORIZED: Not authorized: expected TLS_HANDSHAKE step: SASL_INITIATE ). Query 2b4de6f8564f0cef:9ff7b87500000000 may hang. See IMPALA-2990. I0619 09:55:14.777659 30599 query-state.cc:649] 2b4de6f8564f0cef:9ff7b87500000000] Cancel: query_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:55:44.776489 30597 coordinator-backend-state.cc:472] Cancel query_id= 2b4de6f8564f0cef:9ff7b87500000000 could not do rpc to 10.65.0.101:27000 failure: Cancel() RPC failed: Timed out: connection negotiation to 10.65.0.101:27000 for RPC CancelQueryFInstances timed out after 10.000s (ON_OUTBOUND_QUEUE) I0619 09:55:44.776515 30597 coordinator-backend-state.cc:445] Sending CancelQueryFInstances rpc for query_id=2b4de6f8564f0cef:9ff7b87500000000 backend=10.65.7.41:27000 I0619 09:55:44.786710 30597 coordinator-backend-state.cc:472] Cancel query_id= 2b4de6f8564f0cef:9ff7b87500000000 could not do rpc to 10.65.7.41:27000 failure: Cancel() RPC failed: Not authorized: Client connection negotiation failed: client connection to 10.65.7.41:27000: FATAL_UNAUTHORIZED: Not authorized: expected TLS_HANDSHAKE step: SASL_INITIATE I0619 09:55:44.786741 30597 coordinator.cc:691] CancelBackends() query_id=2b4de6f8564f0cef:9ff7b87500000000, tried to cancel 2 backends I0619 09:55:44.786751 30597 coordinator.cc:863] Release admission control resources for query_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:55:45.789626 30577 impala-beeswax-server.cc:239] close(): query_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:55:45.789664 30577 impala-server.cc:1139] UnregisterQuery(): query_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:55:45.789672 30577 impala-server.cc:1246] Cancel(): query_id=2b4de6f8564f0cef:9ff7b87500000000 I0619 09:55:45.791653 30577 query-exec-mgr.cc:184] ReleaseQueryState(): deleted query_id=2b4de6f8564f0cef:9ff7b87500000000