[HIVE-25912] Drop external table at root of s3 bucket throws NPE

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2, 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Metastore
• Labels: metastore pull-request-available

Environment

Hive version: 3.1.2

• Target Version/s: 3.1.2, 4.0.0

Language

English

• Assignee: Fachuan Bai
• Reporter: Fachuan Bai
• Due: 14/Feb/22
• Created: 30/Jan/22 07:37
• Updated: 17/Nov/22 08:48
• Resolved: 14/Feb/22 06:49

Description

new update: 

I test the master branch, have the same problem.

----------

ENV:

Hive 3.1.2

HDFS:3.3.1

enable OpenLDAP and Ranger .

 

I create the external hive table using this command:

 

CREATE EXTERNAL TABLE `fcbai`(
`inv_item_sk` int,
`inv_warehouse_sk` int,
`inv_quantity_on_hand` int)
PARTITIONED BY (
`inv_date_sk` int) STORED AS ORC
LOCATION
'hdfs://emr-master-1:8020/';

 

The table was created successfully, but  when I drop the table throw the NPE:

 

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.NullPointerException) (state=08S01,code=1)

 

The same bug can reproduction on the other object storage file system, such as S3 or TOS:

CREATE EXTERNAL TABLE `fcbai`(
`inv_item_sk` int,
`inv_warehouse_sk` int,
`inv_quantity_on_hand` int)
PARTITIONED BY (
`inv_date_sk` int) STORED AS ORC
LOCATION
's3a://bucketname/'; // 'tos://bucketname/'

 

I see the source code found:

 common/src/java/org/apache/hadoop/hive/common/FileUtils.java

// check if sticky bit is set on the parent dir
FileStatus parStatus = fs.getFileStatus(path.getParent());
if (!shims.hasStickyBit(parStatus.getPermission())) {
  // no sticky bit, so write permission on parent dir is sufficient
  // no further checks needed
  return;
}

 

because I set the table location to HDFS root path (hdfs://emr-master-1:8020/), so the  path.getParent() function will be return null cause the NPE.

I think have four solutions to fix the bug:

1. modify the create table function, if the location is root dir return create table fail.
2. modify the  FileUtils.checkDeletePermission function, check the path.getParent(), if it is null, the function return, drop successfully.
3. modify the RangerHiveAuthorizer.checkPrivileges function of the hive ranger plugin(in ranger rep), if the location is root dir return create table fail.
4. modify the HDFS Path object, if the URI is root dir, path.getParent() return not null.

I recommend the first or second method, any suggestion for me? thx.

 

 

Issue Links

• links to: GitHub Pull Request #2987
• links to: GitHub Pull Request #3009