[IMPALA-4855] Complex query on large cluster with Kerberos+SSL enabled failed with Couldn't open transport for vd1328.foo:22000 (SASL(-1): generic failure: GSSAPI Error: Invalid token was supplied (Success))

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Not Applicable
• Component/s: Distributed Exec
• Labels: kerberos scalability security
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 01/Feb/17 19:01
• Updated: 14/Nov/18 19:50
• Resolved: 14/Nov/18 19:15

Description

Query failed with

WARNINGS:
Couldn't open transport for vd1328.foo:22000 (SASL(-1): generic failure: GSSAPI Error: Invalid token was supplied (Success))
Couldn't open transport for vd1108.foo:22000 (SASL(-1): generic failure: GSSAPI Error: Invalid token was supplied (Success))
Couldn't open transport for vd1328.foo:22000 (SASL(-1): generic failure: GSSAPI Error: Invalid token was supplied (Success))
Couldn't open transport for vc1514.foo:22000 (SASL(-1): generic failure: GSSAPI Error: Invalid token was supplied (Success))

Query profile attached.
vd1328 has the following errors in the log

I0201 10:40:19.294569 68130 thrift-util.cc:111] TAcceptQueueServer: Caught TException: SSL_accept: error code: 0
I0201 10:40:19.308665 68130 authentication.cc:427] Successfully authenticated principal "impala/vd1113.foo@foo" on an internal connection
I0201 10:40:19.321210 68130 authentication.cc:427] Successfully authenticated principal "impala/vd1113.foo@foo" on an internal connection
I0201 10:40:19.335312 68130 authentication.cc:427] Successfully authenticated principal "impala/vd1109.foo@foo" on an internal connection
I0201 10:40:19.349056 68130 authentication.cc:427] Successfully authenticated principal "impala/vc1515.foo@foo" on an internal connection
I0201 10:40:19.362828 68130 thrift-util.cc:111] TAcceptQueueServer: Caught TException: No more data to read.
I0201 10:40:19.375749 68130 thrift-util.cc:111] TAcceptQueueServer: Caught TException: No more data to read.

On vd1108

I0201 10:37:08.464529 170323 query-exec-mgr.cc:132] GetQueryState(): query_id=e749ebf3e42080ff:4d16bc3f00000000
I0201 10:37:08.476125 170323 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=e749ebf3e42080ff:4d16bc3f00000000 refcnt=34
I0201 10:37:10.015342 189455 status.cc:114] couldn't deserialize thrift msg:
No more data to read.
    @           0x8298e9  (unknown)
    @           0xc6528f  (unknown)
    @           0xc5d3d1  (unknown)
    @           0xc5f0cf  (unknown)
    @           0xc6d288  (unknown)
    @           0xc44704  (unknown)
    @           0xc47ed8  (unknown)
    @           0xc440f2  (unknown)
    @           0xc20f56  (unknown)
    @           0xc22f99  (unknown)
    @           0xbbc3a4  (unknown)
    @           0xbbcd84  (unknown)
    @           0xe08c2a  (unknown)
    @       0x3f0a007aa1  (unknown)
    @       0x3f09ce893d  (unknown)