Hive
  1. Hive
  2. HIVE-3894

Dropping managed HBase table fails

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      HBaseStorageHandler does not delete managed tables from HBase.

      hive> CREATE TABLE foo (bar INT, baz INT) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ('hbase.columns.mapping' = 'f1:baz') TBLPROPERTIES ('hbase.table.name' = 'foo');
      OK
      Time taken: 1.277 seconds
      hive> drop table foo;
      FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.TableNotDisabledException: org.apache.hadoop.hbase.TableNotDisabledException: foo
      	at org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:1396)
      	at org.apache.hadoop.hbase.master.handler.TableEventHandler.<init>(TableEventHandler.java:70)
      	at org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(DeleteTableHandler.java:42)
      	at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1249)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
      	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
      
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:482)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:465)
      	at org.apache.hadoop.hive.hbase.HBaseStorageHandler.commitDropTable(HBaseStorageHandler.java:120)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:599)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
      	at $Proxy9.dropTable(Unknown Source)
      	at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:846)
      	at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:813)
      	at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:3398)
      	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:279)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
      	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1334)
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1120)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      )
      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
      

      The table is gone from Hive metadata though:

      hive> show tables;    
      OK
      Time taken: 0.029 seconds
      

      Attempting to recreate it results in the expected:

      hive> CREATE TABLE foo (bar INT, baz INT) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ('hbase.columns.mapping' = 'f1:baz') TBLPROPERTIES ('hbase.table.name' = 'foo');
      FAILED: Error in metadata: MetaException(message:Table foo already exists within HBase; use CREATE EXTERNAL TABLE instead to register it in Hive.)
      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
      
      1. hbase_managed_table.q.out
        2 kB
        Swarnim Kulkarni
      2. hbase_managed_table.q
        0.4 kB
        Swarnim Kulkarni

        Activity

        Hide
        Swarnim Kulkarni added a comment -

        Nick Dimiduk Emmanuel Guiton I have attached queries and results file as part of my attempt to try and reproduce this. But unfortunately I haven't been able to. I am currently running on hive trunk but it works great for me with hive 0.13 as well. Would you guys mind giving a short with trunk(and or 0.13) and see if this still persists?

        Show
        Swarnim Kulkarni added a comment - Nick Dimiduk Emmanuel Guiton I have attached queries and results file as part of my attempt to try and reproduce this. But unfortunately I haven't been able to. I am currently running on hive trunk but it works great for me with hive 0.13 as well. Would you guys mind giving a short with trunk(and or 0.13) and see if this still persists?
        Swarnim Kulkarni made changes -
        Attachment hbase_managed_table.q [ 12647605 ]
        Attachment hbase_managed_table.q.out [ 12647606 ]
        Hide
        Emmanuel Guiton added a comment -

        Hello,
        I just ran into the same issue.
        I created a table using Hive (0.10) :

        create table is_mappings(key struct<id:int, name: string>, props map<string, string>) 
        stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
        with serdeproperties ("hbase.columns.mapping"=":key, p:") 
        tblproperties ("hbase.table.name"="is_mappings");
        

        Then I ran

        hive> DROP TABLE is_mappings;

        which resulted in

        FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.TableNotDisabledException: org.apache.hadoop.hbase.TableNotDisabledException: is_mappings
                at org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:1508)
                at org.apache.hadoop.hbase.master.handler.TableEventHandler.<init>(TableEventHandler.java:72)
                at org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(DeleteTableHandler.java:47)
                at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1358)
                at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
        
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
                at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
                at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:482)
                at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:465)
                at org.apache.hadoop.hive.hbase.HBaseStorageHandler.commitDropTable(HBaseStorageHandler.java:120)
                at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:599)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
                at com.sun.proxy.$Proxy10.dropTable(Unknown Source)
                at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:841)
                at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:808)
                at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:3372)
                at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:278)
                at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
                at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
                at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
                at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
                at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
                at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
                at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
                at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
                at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
                at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
        )
        FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
        

        When executing

        hive> show tables;

        the table does not appear anymore.
        However, in HBase the table still exists

        hbase(main):010:0* describe 'is_mappings'
        DESCRIPTION                                                                                                                        ENABLED                                                                
         'is_mappings', {NAME => 'p', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSION true                                                                   
         S => '0', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}                                                                                                         
        1 row(s) in 5.1630 seconds
        

        And as stated in the exception, the HBase table is still enabled :

        hbase(main):014:0> drop 'is_mappings'
        
        ERROR: Table is_mappings is enabled. Disable it first.'
        
        Here is some help for this command:
        Drop the named table. Table must first be disabled: e.g. "hbase> drop 't1'"
        
        
        hbase(main):015:0> disable 'is_mappings'
        0 row(s) in 7.0690 seconds
        
        hbase(main):016:0> drop 'is_mappings'
        0 row(s) in 1.1220 seconds
        

        Version of HBase is

        hbase(main):017:0> version
        0.94.10, r1504995, Fri Jul 19 20:24:16 UTC 2013
        
        Show
        Emmanuel Guiton added a comment - Hello, I just ran into the same issue. I created a table using Hive (0.10) : create table is_mappings(key struct<id: int , name: string>, props map<string, string>) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ( "hbase.columns.mapping" = ":key, p:" ) tblproperties ( "hbase.table.name" = "is_mappings" ); Then I ran hive> DROP TABLE is_mappings; which resulted in FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.TableNotDisabledException: org.apache.hadoop.hbase.TableNotDisabledException: is_mappings at org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:1508) at org.apache.hadoop.hbase.master.handler.TableEventHandler.<init>(TableEventHandler.java:72) at org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(DeleteTableHandler.java:47) at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1358) at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96) at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:482) at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:465) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.commitDropTable(HBaseStorageHandler.java:120) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:599) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74) at com.sun.proxy.$Proxy10.dropTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:841) at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:808) at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:3372) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:278) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) ) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask When executing hive> show tables; the table does not appear anymore. However, in HBase the table still exists hbase(main):010:0* describe 'is_mappings' DESCRIPTION ENABLED 'is_mappings', {NAME => 'p', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSION true S => '0', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => ' false ', BLOCKCACHE => ' true '} 1 row(s) in 5.1630 seconds And as stated in the exception, the HBase table is still enabled : hbase(main):014:0> drop 'is_mappings' ERROR: Table is_mappings is enabled. Disable it first.' Here is some help for this command: Drop the named table. Table must first be disabled: e.g. "hbase> drop 't1'" hbase(main):015:0> disable 'is_mappings' 0 row(s) in 7.0690 seconds hbase(main):016:0> drop 'is_mappings' 0 row(s) in 1.1220 seconds Version of HBase is hbase(main):017:0> version 0.94.10, r1504995, Fri Jul 19 20:24:16 UTC 2013
        Hide
        Nick Dimiduk added a comment -

        IIRC, this was Hive 0.10.0 via HDP-1.2.1, but I didn't take notes. git-blame indicates HBaseStorageHandler has support for dropping non-external HBase tables as of HIVE-705.

        Show
        Nick Dimiduk added a comment - IIRC, this was Hive 0.10.0 via HDP-1.2.1, but I didn't take notes. git-blame indicates HBaseStorageHandler has support for dropping non-external HBase tables as of HIVE-705 .
        Hide
        Swarnim Kulkarni added a comment -

        Nick Dimiduk Could you please give more details on the hive and hbase version? I am currently trying to test it out with CDH4.2.1 and hive 0.10 and unable to reproduce what you are seeing. The tables seem to be created and getting dropped properly.

        Show
        Swarnim Kulkarni added a comment - Nick Dimiduk Could you please give more details on the hive and hbase version? I am currently trying to test it out with CDH4.2.1 and hive 0.10 and unable to reproduce what you are seeing. The tables seem to be created and getting dropped properly.
        Swarnim Kulkarni made changes -
        Field Original Value New Value
        Assignee Swarnim Kulkarni [ swarnim ]
        Nick Dimiduk created issue -

          People

          • Assignee:
            Swarnim Kulkarni
            Reporter:
            Nick Dimiduk
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development