[IMPALA-169] DataStreamSender::Channel holds onto client for entire lifetime

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 0.3
• Fix Version/s: Impala 0.7
• Assignee: Skye Wanderman-Milne
• Reporter: Henry Robinson
• Created: 25/Mar/13 23:10
• Updated: 05/Apr/13 16:37
• Resolved: 05/Apr/13 16:37

Description

DataStreamSender::Channel::Init creates a new Thrift client, and then holds on to it for its entire lifetime. This isn't using the client cache appropriately, and can lead to many concurrent socket connections to a backend. Each of those connections means one thread (until we get the non-blocking server fixed), and I've anecdotally seen Impala processes with > 2000 threads, which is 4GB of memory if nothing else.

The right fix is to use the cache when the client is required and not before, and to return the client to the cache when it's done.

(Also the client cache doesn't garbage collect old clients yet, so these sockets stay open forever...)