Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11233

Include Apache Phoenix support in HBaseStorageHandler

    XMLWordPrintableJSON

Details

    • Hide
      Thanks of this patch are possible read or write operations for HBase table in Apache Phoenix persistence mode. Following classes could be considered as 3 keypoints:
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory

      Note: See the code sample provided in description field of that issue for more details.
      Show
      Thanks of this patch are possible read or write operations for HBase table in Apache Phoenix persistence mode. Following classes could be considered as 3 keypoints: - org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory - org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory - org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory Note: See the code sample provided in description field of that issue for more details.
    • Patch

    Description

      Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable key and values. It is necessary when given HBase table is used for persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible with binary sortable format used in Phoenix.

      It turns out the major difference is in all numeric data types accordingly officially provided documentation - https://phoenix.apache.org/language/datatypes.html.

      That's how I use it in my code with applied patch:

          private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) {
              Map<String, String> serdePropertiesMap = new HashMap<>();
      
              serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName());
              serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE);
              serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor));
      
      
              serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName());
              /* Use different key factory for simple and composite primary key */
              if (tableDescriptor.getPkDescriptors().size() == 1) {
                  serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName());
              } else {
                  serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName());
              }
      
              String serDeProperties = serdePropertiesMap.entrySet().stream()
                      .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue()))
                      .collect(Collectors.joining(COLUMNS_SEPARATOR));
      
              logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
              return serDeProperties;
          }
      

      Attachments

        1. HIVE-11233.1.patch
          70 kB
          Svetozar Ivanov
        2. HIVE-11233.2.patch
          69 kB
          Svetozar Ivanov
        3. HIVE-11233.3.patch
          69 kB
          Svetozar Ivanov
        4. HIVE-11233.4.patch
          69 kB
          Svetozar Ivanov
        5. HIVE-11233.5.patch
          69 kB
          Svetozar Ivanov
        6. HIVE-11233.patch
          70 kB
          Svetozar Ivanov
        7. HIVE-11233-branch-1.2.patch
          71 kB
          Svetozar Ivanov
        8. HIVE-11233-branch-2.0.patch
          70 kB
          Svetozar Ivanov

        Activity

          People

            svetozari Svetozar Ivanov
            svetozari Svetozar Ivanov
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: