[CASSANDRA-3142] CustomTThreadPoolServer should log TTransportException at DEBUG level

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 0.8.6
• Severity: Low
• Assignee: James P. Ancona
• Reporter: James P. Ancona

Authors

James P. Ancona

Reviewers

Pavel Yaskevich

• Created: 06/Sep/11 02:10
• Updated: 16/Apr/19 09:32
• Resolved: 07/Sep/11 13:42

Description

Currently CustomTThreadPoolServer, like the Thrift TThreadPoolServer, silently ignores TTransportException in its run() method. This is appropriate in most cases because TTransportException occurs fairly often when client connections die. However TTransportException is also thrown when TFramedTransport encounters a frame that is larger than thrift_framed_transport_size_in_mb. In that case, silently exiting the run loop leads to a SocketException on the client side which can be both difficult to diagnose, in part because nothing is logged by Cassandra, and high-impact, because the client may respond by marking the server node down and retrying the too-large request on another node, where it also fails. This process repeated leads to the entire cluster being marked down (see https://github.com/rantav/hector/issues/212). I've filed two Thrift issues (https://issues.apache.org/jira/browse/THRIFT-1323 and https://issues.apache.org/jira/browse/THRIFT-1324), but in the meantime, I suggest that CustomTThreadPoolServer log the exception at DEBUG level in order to support easier troubleshooting.