[THRIFT-2682] TThreadedServer leaks per-thread memory

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 0.9.1
• Fix Version/s: 0.9.3
• Component/s: C++ - Library

Environment

RHEL 6.5

• Assignee: James E. King III
• Reporter: Artem Koval
• Created: 29/Aug/14 11:03
• Updated: 21/Jul/15 02:21
• Resolved: 06/May/15 00:02

Description

We are using Thrift 0.9.1 in Linux, both client and server are written in C++. TThreadedServer is used to process requests involving large (up to 1MB) binary parameters.

Usage pattern is like this: the client establishes a UNIX socket connection, makes some interface calls and then disconnects.

We noticed that the memory footprint of the server process increases over time. Valgrind inspection showed that despite the server threads have long been shutdown, their data allocated in TThreadedServer::serve() hasn't been freed and shows up as "definitely lost".

This leak, although significant, is not critical by itself. But per-thread data includes a TBinaryProtocol object, which in turn holds a "string buffer" used to serialize binary parameters. These string buffers are quite large in our case, leading to overall memory leakage being unacceptably high.

As a workaround, we will for now turn to TNonblockingServer, which uses persistent threads. However, a memory leak is still a very serious problem, and it should be fixed.