[SENTRY-654] Calls to append_partition fail when Sentry is enabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0

Environment

CDH 5.3.1

• Assignee: Ryan P
• Reporter: Richard Williams
• Created: 14/Feb/15 00:23
• Updated: 22/Apr/15 16:16
• Resolved: 17/Mar/15 23:20

Description

When the Hive Metastore has org.apache.sentry.binding.metastore.MetastoreAuthzBinding configured as a pre event listener class, calls to the metastore Thrift API's append_partition method fail with a MetaException whose message is "java.lang.NullPointerException". In practice, this causes LOAD DATA INPATH statements targeting particular partitions of a partitioned table to fail. Example stack trace:

Exception in thread "main" MetaException(message:java.lang.NullPointerException)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$append_partition_with_environment_context_result$append_partition_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41079)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$append_partition_with_environment_context_result$append_partition_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41038)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$append_partition_with_environment_context_result.read(ThriftHiveMetastore.java:40961)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_append_partition_with_environment_context(ThriftHiveMetastore.java:1355)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.append_partition_with_environment_context(ThriftHiveMetastore.java:1339)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.appendPartition(HiveMetaStoreClient.java:488)
at Test.main(Test.java:31)

The likely cause of this is that HiveMetastoreAuthzBinding's authorizeAddPartition method, which is triggered on both add_partition calls and append_partition calls, assumes that the added Partition object's storage descriptor is not null. However, org.apache.hadoop.hive.metastore.HiveMetaStore's append_partition_common method creates a Partition object on the fly based on the provided database, table, and partition values, and it doesn't initialize that object's storage descriptor until after it has already called firePreEvent (thereby signaling the pre event listeners to execute).

Issue Links

• relates to: SENTRY-703 Calls to add_partition fail when passed a Partition object with a null location