[IMPALA-11548] TTransportException can be thrown for RPC connection that is open for long time

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.2.0
• Fix Version/s: Impala 4.2.0
• Component/s: Backend
• Epic Color: ghx-label-4
• Assignee: Riza Suminto
• Reporter: Riza Suminto
• Created: 31/Aug/22 13:09
• Updated: 07/Sep/22 08:12
• Resolved: 07/Sep/22 06:52

Description

After Impala upgrade CPP thrift version to 0.16.0, we are seeing TTransportException thrown for RPC connection that is open for long time.

thrift-0.16.0 add counter to limit message size at https://github.com/apache/thrift/commit/63213c1.
The TTransportException is thrown when remainingMessageSize < numBytes, indicating that TTransport was not flushed properly.
This is presumably due to the fact that both input_ and output_ TProtocol within TAcceptQueueServer::Task has different TTransport object within them. Letting both input_ and output_to share the same TTransport object might fix the flushing mechanism and properly reset the limit counter of the shared TTransport.

See:
https://github.com/apache/impala/blob/fec7a79c50de9ef5db2c652ece111aaa303d57c7/be/src/transport/TSaslServerTransport.cpp#L132-L146