[IMPALA-5394] Set socket timeouts while opening TSaslTransport

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.11.0
• Component/s: Distributed Exec
• Environment: Kerberos
• Target Version: Impala 2.9.0
• Epic Color: ghx-label-9
• Assignee: John Sherman
• Reporter: Doug Cameron
• Created: 31/May/17 02:53
• Updated: 03/Oct/18 21:08
• Resolved: 05/Oct/17 05:39

Description

Similar to IMPALA-3875, on a kerberized system, when the HS2 server does the initial SASL negotiation after the open, if the client never sends any data, the read() will hang and hangs the entire server port.

In detail:

- TThreadPoolServer calls getTransport() on a client from the Server
      thread (the thread that does the accepts).
      - TSaslServerTransport->getTransport() calls TSaslTransport->open()
      - TSaslServerTransport->open() tries to negotiate SASL which calls
        read/write
        - If read/write blocks, the TThreadPoolServer cannot accept
          connections
    - This can be demonstrated by running against a kerberos enabled cluster:
            nc <impala host> <hs2 port> &
      then trying to connect to the hs2 port via beeline. The beeline
      connection will hang until the nc process is killed.
    - Can fix by setting the underlying TSocket recvTimeout and sendTimeout
      before the TSaslServerTransport->open() and reset them to 0 after
      open() completes.
    - Consider adding sasl_connect_tcp_timeout_seconds command line option (defaults to 10, 0 to disable)

Issue Links

• duplicates: IMPALA-7566 TAcceptQueueServer connection setup should have timeout
• is duplicated by: IMPALA-5268 After canceling query on secure cluster coordinator node doesn't accept new connections
• is duplicated by: IMPALA-1941 TSaslServerTransport blocked by misbehaving clients
• is related to: IMPALA-7638 Lower default timeout for connection setup
• is related to: IMPALA-5268 After canceling query on secure cluster coordinator node doesn't accept new connections
• Parent Feature: IMPALA-3380 Add TCP timeouts to all RPCs that don't block