[HIVE-19424] NPE In MetaDataFormatters

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Component/s: HiveServer2, Metastore, Standalone Metastore
• Release Note: HIVE-19424 : Fixing NPE In MetaDataFormatters at Branch-3
• Assignee: Alice Fan
• Reporter: David Mollitor
• Created: 04/May/18 19:38
• Updated: 17/Nov/22 08:54
• Resolved: 24/Jul/18 14:02

Description

Overview

According to the Hive Schema definition, a table's INPUT_FORMAT class can be set to NULL. However, there are places in the code where we do not account for this NULL value, in particular the MetaDataFormatters classes TextMetaDataFormatter and JsonMetaDataFormatter. In addition, there is no debug level logging in the MetaDataFormatters classes to tell me which table in particular is causing the problem.

hive-schema-2.2.0.mysql.sql

CREATE TABLE IF NOT EXISTS `SDS` (
  `SD_ID` bigint(20) NOT NULL,
  `CD_ID` bigint(20) DEFAULT NULL,
  `INPUT_FORMAT` varchar(4000) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
  `IS_COMPRESSED` bit(1) NOT NULL,
...

TextMetaDataFormatter.java

// Not checking for a null return from getInputFormatClass
inputFormattCls = par.getInputFormatClass().getName();
outputFormattCls = par.getOutputFormatClass().getName();

Reproduction

-- MySQL Backend
update SDS SET INPUT_FORMAT=NULL WHERE SD_ID=XXX;

// Hive
SHOW TABLE EXTENDED FROM default LIKE '*';

// HS2 Logs
[HiveServer2-Background-Pool: Thread-464]: Error running hive query: 
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table status
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
	at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
	at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
	at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Exception while processing show table status
	at org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:3025)
	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:405)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
	... 11 more
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.metadata.formatting.TextMetaDataFormatter.showTableStatus(TextMetaDataFormatter.java:202)
	at org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:3020)
	... 20 more

Issue Links

• is related to: IMPALA-6985 Impala View Does Not Populate Table Input/Output Format Class