[THRIFT-970] Under heavy load, THttpClient may fail with "too many open files"

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2, 0.3, 0.4, 0.5
• Fix Version/s: 0.6
• Component/s: Java - Library

Environment

All

• Assignee: Mathias Herberts
• Reporter: Mathias Herberts
• Created: 25/Oct/10 09:43
• Updated: 08/Nov/10 17:22
• Resolved: 05/Nov/10 17:16

Description

THttpClient uses URL.openConnection which returns a HttpUrlConnection instance for each message transmission.

HttpUrlConnection supposedly pools connections to the server. While stress testing an application, we've noticed that after several thousands requests, THttpClient would fail with a "Too many open files" error thrown in java.net.Socket.createImpl called from THttpClient.flush.

As the underlying connection to the server is internally handled by HttpUrlConnection and the JVM, there is unfortunately not much that can be done to remedy this problem while still using HttpUrlConnection.

I propose a new implementation of THttpClient which uses Apache HttpClient from Http Components instead of HttpUrlConnection.