Hive
  1. Hive
  2. HIVE-1634

Allow access to Primitive types stored in binary format in HBase

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0, 0.8.0, 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      This addresses HIVE-1245 in part, for atomic or primitive types.

      The serde property "hbase.columns.storage.types" = ",b,b,b,b,b,b,b,b" is a specification of the storage option for the corresponding column in the serde property "hbase.columns.mapping". Allowed values are '' for table default, 's' for standard string storage, and 'b' for binary storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families use a colon separated pair such as 's:b' for the key and value part specifiers respectively. See the test cases and queries for HBase handler for additional examples.

      There is also a table property "hbase.table.default.storage.type" = "string" to specify a table level default storage type. The other valid specification is "binary". The table level default is overridden by a column level specification.

      This control is available for the boolean, tinyint, smallint, int, bigint, float, and double primitive types. The attached patch also relaxes the mapping of map types to HBase column families to allow any primitive type to be the map key.

      Attached is a program for creating a table and populating it in HBase. The external table in Hive can access the data as shown in the example below.

      hive> create external table TestHiveHBaseExternalTable
      > (key string, c_bool boolean, c_byte tinyint, c_short smallint,
      > c_int int, c_long bigint, c_string string, c_float float, c_double double)
      > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
      > with serdeproperties ("hbase.columns.mapping" = ":key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double")
      > tblproperties ("hbase.table.name" = "TestHiveHBaseExternalTable");
      OK
      Time taken: 0.691 seconds
      hive> select * from TestHiveHBaseExternalTable;
      OK
      key-1 NULL NULL NULL NULL NULL Test-String NULL NULL
      Time taken: 0.346 seconds
      hive> drop table TestHiveHBaseExternalTable;
      OK
      Time taken: 0.139 seconds
      hive> create external table TestHiveHBaseExternalTable
      > (key string, c_bool boolean, c_byte tinyint, c_short smallint,
      > c_int int, c_long bigint, c_string string, c_float float, c_double double)
      > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
      > with serdeproperties (
      > "hbase.columns.mapping" = ":key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double",
      > "hbase.columns.storage.types" = "-,b,b,b,b,b,b,b,b" )
      > tblproperties (
      > "hbase.table.name" = "TestHiveHBaseExternalTable",
      > "hbase.table.default.storage.type" = "string");
      OK
      Time taken: 0.139 seconds
      hive> select * from TestHiveHBaseExternalTable;
      OK
      key-1 true -128 -32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291
      Time taken: 0.151 seconds
      hive> drop table TestHiveHBaseExternalTable;
      OK
      Time taken: 0.154 seconds
      hive> create external table TestHiveHBaseExternalTable
      > (key string, c_bool boolean, c_byte tinyint, c_short smallint,
      > c_int int, c_long bigint, c_string string, c_float float, c_double double)
      > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
      > with serdeproperties (
      > "hbase.columns.mapping" = ":key,cf:boolean,cf:byte,cf:short,cf:int,cf:long,cf:string,cf:float,cf:double",
      > "hbase.columns.storage.types" = ",b,b,b,b,b,,b,b" )
      > tblproperties ("hbase.table.name" = "TestHiveHBaseExternalTable");
      OK
      Time taken: 0.347 seconds
      hive> select * from TestHiveHBaseExternalTable;
      OK
      key-1 true -128 -32768 -2147483648 -9223372036854775808 Test-String -2.1793132E-11 2.01345E291
      Time taken: 0.245 seconds
      hive>

      1. HIVE-1634.branch08.patch
        247 kB
        Alan Gates
      2. hive-1634_3.patch
        247 kB
        Ashutosh Chauhan
      3. ASF.LICENSE.NOT.GRANTED--HIVE-1634.D1581.3.patch
        246 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--HIVE-1634.D1581.2.patch
        247 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--HIVE-1634.D1581.1.patch
        243 kB
        Phabricator
      6. HIVE-1634.1.patch
        249 kB
        John Sichi
      7. HIVE-1634.0.patch
        561 kB
        Basab Maulik
      8. TestHiveHBaseExternalTable.java
        2 kB
        Basab Maulik

        Issue Links

          Activity

          Basab Maulik created issue -
          Basab Maulik made changes -
          Field Original Value New Value
          Attachment TestHiveHBaseExternalTable.java [ 12454483 ]
          Attachment HIVE-1634.0.patch [ 12454484 ]
          Basab Maulik made changes -
          Link This issue blocks HIVE-1245 [ HIVE-1245 ]
          John Sichi made changes -
          Attachment HIVE-1634.1.patch [ 12467588 ]
          Phabricator made changes -
          Attachment HIVE-1634.D1581.1.patch [ 12513042 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.8.0 [ 12316178 ]
          Affects Version/s 0.9.0 [ 12317742 ]
          Assignee Basab Maulik [ bkm ] Ashutosh Chauhan [ ashutoshc ]
          Phabricator made changes -
          Attachment HIVE-1634.D1581.2.patch [ 12513545 ]
          Ashutosh Chauhan made changes -
          Link This issue blocks HIVE-2815 [ HIVE-2815 ]
          Carl Steinbach made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Phabricator made changes -
          Attachment HIVE-1634.D1581.3.patch [ 12517355 ]
          Ashutosh Chauhan made changes -
          Attachment hive-1634_3.patch [ 12517360 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.9.0 [ 12317742 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Alan Gates made changes -
          Attachment HIVE-1634.branch08.patch [ 12518546 ]
          Ashutosh Chauhan made changes -
          Issue Type Improvement [ 4 ] New Feature [ 2 ]
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Basab Maulik
            • Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development