[IMPALA-6720] impala deamon server 22000 connect refuse in cluster running time

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.11.0
• Component/s: Backend
• Environment: centos 7
• Flags: Important
• Target Version: Product Backlog
• Epic Color: ghx-label-7
• Assignee: Unassigned
• Reporter: wondering
• Created: 22/Mar/18 08:44
• Updated: 23/Dec/20 21:36
• Resolved: 23/Dec/20 21:36

Description

  I0322 08:18:37.156484 52026 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=e14c21df404af739:8ccb93bf00000000 refcnt=5
I0322 08:18:38.122870 52029 scan-node.cc:176] Not all filters arrived (arrived: [], missing [1]), waited for 462ms
I0322 08:18:38.369796 17017 thrift-util.cc:123] TSocket::read() recv() <Host: * * * * Port: 22065>Connection reset by peer
I0322 08:18:38.369956 17017 thrift-util.cc:123] TAcceptQueueServer client died: ECONNRESET
I0322 08:18:38.369989 52034 client-cache.cc:81] ReopenClient(): re-creating client for * * * *:22000
I0322 08:18:38.369990 49723 thrift-util.cc:123] TSocket::read() recv() <Host: * * * * Port: 59634>Connection reset by peer
I0322 08:18:38.370007 46084 thrift-util.cc:123] TSocket::read() recv() <Host: * * * * Port: 56998>Connection reset by peer
I0322 08:18:38.370066 49723 thrift-util.cc:123] TAcceptQueueServer client died: ECONNRESET
I0322 08:18:38.370151 46086 thrift-util.cc:123] TSocket::read() recv() <Host: * * * * Port: 57000>Connection reset by peer
I0322 08:18:38.370170 45833 thrift-util.cc:123] TSocket::read() recv() <Host: * * * * Port: 33499>Connection reset by peer
I0322 08:18:38.370227 45833 thrift-util.cc:123] TAcceptQueueServer client died: ECONNRESET
I0322 08:18:38.370277 46084 thrift-util.cc:123] TAcceptQueueServer client died: ECONNRESET
I0322 08:18:38.370316 46086 thrift-util.cc:123] TAcceptQueueServer client died: ECONNRESET
I0322 08:18:38.370590 52034 thrift-util.cc:123] TSocket::open() connect() <Host: * * * * Port: 22000>Connection refused
I0322 08:18:38.370775 52034 thrift-client.cc:77] Couldn't open transport for * * * *:22000 (connect() failed: Connection refused)
I0322 08:18:38.370805 52034 thrift-client.cc:93] Unable to connect to * * * *:22000
I0322 08:18:38.370815 52034 thrift-client.cc:99] (Attempt 1 of 3)
I0322 08:18:38.371094 52034 thrift-util.cc:123] TSocket::open() connect() <Host: * * * * Port: 22000>Connection refused
I0322 08:18:38.371160 52034 thrift-client.cc:77] Couldn't open transport for * * * *:22000 (connect() failed: Connection refused)
I0322 08:18:38.371199 52034 thrift-client.cc:93] Unable to connect to * * * *:22000
I0322 08:18:38.371204 52034 thrift-client.cc:99] (Attempt 2 of 3)
I0322 08:18:38.371469 52034 thrift-util.cc:123] TSocket::open() connect() <Host: * * * * Port: 22000>Connection refused
I0322 08:18:38.371526 52034 thrift-client.cc:77] Couldn't open transport for * * * *:22000 (connect() failed: Connection refused)
I0322 08:18:38.371532 52034 thrift-client.cc:93] Unable to connect to * * * *:22000
I0322 08:18:38.371536 52034 thrift-client.cc:99] (Attempt 3 of 3)
I0322 08:18:38.371551 52034 client-cache.h:329] RPC client failed to connect: Couldn't open transport for * * * *:22000 (connect() failed: Connection refused)
rpc: N6impala19TTransmitDataResultE
I0322 08:18:38.371559 52034 client-cache.cc:170] Broken Connection, destroy client for * * * *:22000
E0322 08:18:38.371644 52030 data-stream-sender.cc:278] channel send status: RPC client failed to connect: Couldn't open transport for * * * *:22000 (connect() failed: Connection refused)
 
I0322 08:18:38.372494 45854 impala-internal-service.cc:63] CancelQueryFInstances(): query_id=ec4256d5c8efc133:aba62f4700000000
I0322 08:18:38.372546 45854 query-exec-mgr.cc:92] QueryState: query_id=ec4256d5c8efc133:aba62f4700000000 refcnt=3

Issue Links

• relates to: IMPALA-2567 KRPC milestone 1
• relates to: IMPALA-5020 Query against large cluster with SSL + Kerberos enabled failed with RPC client failed to connect: Couldn't open transport for foo:22000 (Could not resolve host for client socket.)
• relates to: IMPALA-3189 Address scalability issue with N^2 KDC requests on cluster startup