[HIVE-23751] QTest: Override #mkdirs() method in ProxyFileSystem To Align After HADOOP-16582

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: Syed Shameerur Rahman
• Reporter: Syed Shameerur Rahman
• Created: 23/Jun/20 10:05
• Updated: 17/Nov/22 08:47
• Resolved: 01/Jul/20 16:41

Description

HADOOP-16582 have changed the way how mkdirs() work:

Before HADOOP-16582:
All calls to mkdirs(p) were fast-tracked to FileSystem.mkdirs which were then re-routed to mkdirs(p, permission) method. For ProxyFileSytem the call would look like

FileUtiles.mkdir(p)  ----->  FileSystem.mkdirs(p) ---> ProxyFileSytem.mkdirs(p,permission)

An implementation of FileSystem have only needed implement mkdirs(p, permission)

After HADOOP-16582:

Since FilterFileSystem overrides mkdirs(p) method the new call to ProxyFileSystem would look like

FileUtiles.mkdir(p) ---> FilterFileSystem.mkdirs(p) -->

This will make all the qtests fails with the below exception

Caused by: java.lang.IllegalArgumentException: Wrong FS: pfile:/media/ebs1/workspace/hive-3.1-qtest/group/5/label/HiveQTest/hive-1.2.0/itests/qtest/target/warehouse/dest1, expected: file:///

Note: We will hit this issue when we bump up hadoop version in hive.

So as per the discussion in HADOOP-16963 ProxyFileSystem would need to override the mkdirs(p) method inorder to solve the above problem. So now the new flow would look like

FileUtiles.mkdir(p)  ---->   ProxyFileSytem.mkdirs(p) ---> ProxyFileSytem.mkdirs(p, permission) --->

Issue Links

• is caused by: HADOOP-16582 LocalFileSystem's mkdirs() does not work as expected under viewfs.
• relates to: HADOOP-16963 HADOOP-16582 changed mkdirs() behavior
• links to: GitHub Pull Request #1167