[HADOOP-19279] ABFS: Disabling Apache Http Client as Default Http Client for ABFS Driver

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.4.1
• Fix Version/s: 3.5.0, 3.4.1
• Component/s: fs/azure
• Labels: pull-request-available
• Assignee: Manika Joshi
• Reporter: Manika Joshi
• Created: 18/Sep/24 17:46
• Updated: 24/Sep/24 20:03
• Resolved: 24/Sep/24 13:04

Description

As part of work done under HADOOP-19120 [ABFS]: ApacheHttpClient adaptation as network library - ASF JIRA
Apache Http Client was introduced as an alternative Network Library that can be used with ABFS Driver. Earlier JDK Http Client was the only supported network library.

Apache Http Client was found to be more helpful in terms of controls and knobs it provides to better manage the Network aspects of driver. Hence it was made the default Network Client to be used with ABFS Driver.

Recently while running scale workloads, we observed a regression where some unexpected wait time was observed while establishing connections. A possible fix has been identified and we are working on getting it fixed.
One scenario identified during internal tests revealed that, with the current Apache client implementation, a connection cannot become stale. As a result, it is safe to remove the check for stale connections when closing them. This change will optimize the connection handling process by eliminating unnecessary delays and reducing the risk of potential failures caused by redundant checks.
There was also a possible NPE scenario which was identified on the new network client code recently.

Until we are done with the code fixes and have revalidated the whole Apache client flow, we would like to make JDK Client as default client again. The new support for Apache Http Client will still be there, but it will be disabled behind a config.

Issue Links

• links to: GitHub Pull Request #7055