[HADOOP-19120] [ABFS]: ApacheHttpClient adaptation as network library

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0, 3.4.1
• Component/s: fs/azure
• Labels: pull-request-available

Release Note

Apache httpclient 4.5.x is a new implementation of http connections; this supports a large configurable pool of connections along with the ability to limit their lifespan.

The networking library can be chosen using the configuration
option fs.azure.networking.library

The supported values are
- JDK_HTTP_URL_CONNECTION : Use JDK networking library [Default]
- APACHE_HTTP_CLIENT : Use Apache HttpClient

Important: when the networking library is switched to
the Apache http client, the apache httpcore and httpclient must be on the classpath.

• Assignee: Pranav Saxena
• Reporter: Pranav Saxena
• Created: 21/Mar/24 10:49
• Updated: 24/Sep/24 13:04
• Resolved: 22/Jul/24 18:06

Description

Apache HttpClient is more feature-rich and flexible and gives application more granular control over networking parameter.

ABFS currently relies on the JDK-net library. This library is managed by OpenJDK and has no performance problem. However, it limits the application's control over networking, and there are very few APIs and hooks exposed that the application can use to get metrics, choose which and when a connection should be reused. ApacheHttpClient will give important hooks to fetch important metrics and control networking parameters.

A custom implementation of connection-pool is used. The implementation is adapted from the JDK8 connection pooling. Reasons for doing it:
1. PoolingHttpClientConnectionManager heuristic caches all the reusable connections it has created. JDK's implementation only caches limited number of connections. The limit is given by JVM system property "http.maxConnections". If there is no system-property, it defaults to 5. Connection-establishment latency increased with all the connections were cached. Hence, adapting the pooling heuristic of JDK netlib,
2. In PoolingHttpClientConnectionManager, it expects the application to provide `setMaxPerRoute` and `setMaxTotal`, which the implementation uses as the total number of connections it can create. For application using ABFS, it is not feasible to provide a value in the initialisation of the connectionManager. JDK's implementation has no cap on the number of connections it can have opened on a moment. Hence, adapting the pooling heuristic of JDK netlib,

Issue Links

• causes: HADOOP-19271 [ABFS]: NPE in AbfsManagedApacheHttpConnection.toString() when not connected
• links to: GitHub Pull Request #6633
• links to: GitHub Pull Request #6959