Details

    • Release Note:
      HIVE-19424 : Fixing NPE In MetaDataFormatters at Branch-3

      Description

      Overview

      According to the Hive Schema definition, a table's INPUT_FORMAT class can be set to NULL. However, there are places in the code where we do not account for this NULL value, in particular the MetaDataFormatters classes TextMetaDataFormatter and JsonMetaDataFormatter. In addition, there is no debug level logging in the MetaDataFormatters classes to tell me which table in particular is causing the problem.

      hive-schema-2.2.0.mysql.sql
      CREATE TABLE IF NOT EXISTS `SDS` (
        `SD_ID` bigint(20) NOT NULL,
        `CD_ID` bigint(20) DEFAULT NULL,
        `INPUT_FORMAT` varchar(4000) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
        `IS_COMPRESSED` bit(1) NOT NULL,
      ...
      
      TextMetaDataFormatter.java
      // Not checking for a null return from getInputFormatClass
      inputFormattCls = par.getInputFormatClass().getName();
      outputFormattCls = par.getOutputFormatClass().getName();
      

      Reproduction

      -- MySQL Backend
      update SDS SET INPUT_FORMAT=NULL WHERE SD_ID=XXX;
      
      // Hive
      SHOW TABLE EXTENDED FROM default LIKE '*';
      
      // HS2 Logs
      [HiveServer2-Background-Pool: Thread-464]: Error running hive query: 
      org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table status
      	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
      	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
      	at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
      	at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
      	at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Exception while processing show table status
      	at org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:3025)
      	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:405)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
      	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
      	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
      	... 11 more
      Caused by: java.lang.NullPointerException
      	at org.apache.hadoop.hive.ql.metadata.formatting.TextMetaDataFormatter.showTableStatus(TextMetaDataFormatter.java:202)
      	at org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:3020)
      	... 20 more
      

        Attachments

        1. HIVE-19424.1.patch
          3 kB
          Alice Fan
        2. HIVE-19424.2.patch
          3 kB
          Alice Fan
        3. HIVE-19424.3.patch
          3 kB
          Alice Fan
        4. HIVE-19424.3-branch-3.patch
          3 kB
          Yongzhi Chen

          Issue Links

            Activity

              People

              • Assignee:
                afan Alice Fan
                Reporter:
                belugabehr David Mollitor
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: