[HADOOP-18445] Migrate to new azure-storage SDKs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs/azure
• Assignee: Unassigned
• Reporter: Yaniv Kunda
• Created: 07/Sep/22 11:42
• Updated: 07/Sep/22 11:48

Description

The last time the azure-storage sdk was updated was more than 2 years ago:
https://github.com/apache/hadoop/blame/trunk/hadoop-project/pom.xml#L1545

The sdk have since been updated and have gone through major design changes, the reasoning for which are documented here:
https://github.com/Azure/azure-storage-java/blob/master/V12%20Upgrade%20Story.md

Upgrading to the latest sdk will bring many improvements, performance improvements and bug fixes - too many to list or count.
In order the to move forward with time, and avoid being stuck when the service API versions start getting deprecated and removed, this needs to be addressed.

The needed changes would be mostly in the following modules:

1. https://github.com/apache/hadoop/tree/trunk/hadoop-tools/hadoop-azure
2. https://github.com/apache/hadoop/tree/trunk/hadoop-tools/hadoop-azure-datalake