[IMPALA-5020] Query against large cluster with SSL + Kerberos enabled failed with RPC client failed to connect: Couldn't open transport for foo:22000 (Could not resolve host for client socket.)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.8.0
• Component/s: Distributed Exec
• Labels: scalability
• Assignee: Sailesh Mukil
• Reporter: Mostafa Mokhtar
• Created: 03/Mar/17 01:00
• Updated: 22/Mar/18 23:41
• Resolved: 30/Aug/17 20:30

Description

None of the Impalads crashed, yet there is the following errors in the log

I0302 08:28:33.072108 123308 client-cache.h:260] client 0x3fb77e40 unexpected exception: SSL_read: Resource temporarily unavailable, type=N6apache6thrift9transport13TSSLExceptionE
I0302 08:28:33.072561 123308 client-cache.cc:81] ReopenClient(): re-creating client for va1035foo.com:22000
I0302 08:28:33.075028 123232 client-cache.h:260] client 0x7f8a9cda2a40 unexpected exception: SSL_read: Resource temporarily unavailable, type=N6apache6thrift9transport13TSSLExceptionE
I0302 08:28:33.075036 123261 client-cache.h:260] client 0x7f8a5a2929c0 unexpected exception: SSL_read: Resource temporarily unavailable, type=N6apache6thrift9transport13TSSLExceptionE
I0302 08:28:33.075350 123232 client-cache.cc:81] ReopenClient(): re-creating client for ve1125foo.com:22000
I0

I0302 08:33:58.108232 123247 thrift-util.cc:111] TSocket::open() getaddrinfo() <Host: vc1523foo.com Port: 22000>Temporary failure in name resolution
I0302 08:33:58.110311 123247 status.cc:114] Couldn't open transport for vc1523foo.com:22000 (Could not resolve host for client socket.)
    @           0x837d39  (unknown)
    @           0xda01a6  (unknown)
    @           0xda04c2  (unknown)
    @           0xa4ce4b  (unknown)
    @           0xa4d76a  (unknown)
    @           0xa69ca8  (unknown)
    @           0xdc5f8e  (unknown)
    @           0xdc2a8d  (unknown)
    @           0xdc3c42  (unknown)
    @           0xdc4001  (unknown)
    @           0xdc4dcd  (unknown)
    @           0xbd4149  (unknown)
    @           0xbd4b24  (unknown)
    @           0xe1ab8a  (unknown)
    @       0x3a6da07aa1  (unknown)
    @       0x3a6d6e893d  (unknown)
I0302 08:33:58.110528 123247 thrift-client.cc:67] Unable to connect to vc1523foo.com:22000
I0302 08:33:58.110704 123247 thrift-client.cc:73] (Attempt 1 of 3)
I0302 08:34:03.117561 123287 thrift-util.cc:111] TSocket::open() getaddrinfo() <Host: vd1115foo.com Port: 22000>Temporary failure in name resolution
I0302 08:34:03.119623 123287 status.cc:114] Couldn't open transport for vd1115foo.com:22000 (Could not resolve host for client socket.)
    @           0x837d39  (unknown)

Issue Links

• duplicates: IMPALA-3189 Address scalability issue with N^2 KDC requests on cluster startup
• is duplicated by: IMPALA-3189 Address scalability issue with N^2 KDC requests on cluster startup
• is related to: IMPALA-6720 impala deamon server 22000 connect refuse in cluster running time