[ZOOKEEPER-3573] Dealing with long TLS connection closing time without SO_LINGER option

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.6.0
• Component/s: server
• Assignee: Unassigned
• Reporter: Jie Huang
• Created: 12/Oct/19 22:49
• Updated: 20/Dec/19 17:41

Description

As described in ZOOKEEPER-3384, with SSL sockets, a close_notify is required to be sent before closing the write side of a connection. When the send buffer is full and the writing is blocked, it will take a long time to send close_notify thus a long time to close the socket. The long closing time on followers with a partitioned-away leader would stall the shutdown process and delay a new leader election to establish a new quorum. As a result, the ensemble would be unavailable for a long time.

In ZOOKEEPER-3384, SO_LINGER option is used to close the socket quickly (and potentially uncleanly). In JDK 11, however, SO_LINGER option is not honored so we need a new way to avoid the long quorum unavailable time.

Sub-Tasks

• Close quorum socket asynchronously to avoid server shutdown stalled by long socket closing time: Closed
• Moving sending packets in Learner to a separate thread: Closed
• Close quorum socket asynchronously on the leader to avoid ping being blocked by long socket closing time: Resolved
• learner.asyncSending, learner.closeSocketAsync and leader.closeSocketAsync should be configurable in zoo.cfg: Resolved