[HADOOP-17089] WASB: Update azure-storage-java SDK

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0, 3.2.0
• Fix Version/s: 2.10.1, 3.3.1
• Component/s: fs/azure
• Release Note: Azure WASB bug fix that can cause list results to appear empty.
• Assignee: Thomas Marqardt
• Reporter: Thomas Marqardt
• Created: 24/Jun/20 13:14
• Updated: 11/Nov/20 01:46
• Resolved: 25/Jun/20 05:51

Description

WASB depends on the Azure Storage Java SDK.  There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty.  This causes the Filesystem listStatus and similar APIs to return empty results.  This has been seen in Spark work loads when jobs use more than one executor core. 

See https://github.com/Azure/azure-storage-java/pull/546 for details on the bug in the Azure Storage SDK.

This issue can cause data loss.

Issue Links

• relates to: BIGTOP-3444 Bump Hadoop to 2.10.1
• links to: PR 2099