[HADOOP-15872] ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.1
• Component/s: fs/azure
• Assignee: junhua gu
• Reporter: Thomas Marqardt
• Created: 22/Oct/18 21:52
• Updated: 28/Nov/18 17:24
• Resolved: 23/Nov/18 14:24

Description

This update to the latest REST version (2018-11-09) will make the following changes to the ABFS driver:

1) The ABFS implementation of getFileStatus currently requires read permission.  According to HDFS permissions guide, it should only require execute on the parent folders (traversal access).  A new REST API has been introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem.

2) The new "2018-11-09" REST version introduces support to i) automatically translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) optionally translate OIDs to UPNs in the responses when getting the owner, owning group, or ACL.  Configuration will be introduced to optionally translate OIDs to UPNs in the responses.  Since translation has a performance impact, the default will be to perform no translation and return the OIDs.

Issue Links

• blocks: HADOOP-15812 ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console
• blocks: HADOOP-15876 Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore