Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11233

Include Apache Phoenix support in HBaseStorageHandler

    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Release Note:
      Hide
      Thanks of this patch are possible read or write operations for HBase table in Apache Phoenix persistence mode. Following classes could be considered as 3 keypoints:
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory

      Note: See the code sample provided in description field of that issue for more details.
      Show
      Thanks of this patch are possible read or write operations for HBase table in Apache Phoenix persistence mode. Following classes could be considered as 3 keypoints: - org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory - org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory - org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory Note: See the code sample provided in description field of that issue for more details.
    • Flags:
      Patch

      Description

      Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable key and values. It is necessary when given HBase table is used for persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible with binary sortable format used in Phoenix.

      It turns out the major difference is in all numeric data types accordingly officially provided documentation - https://phoenix.apache.org/language/datatypes.html.

      That's how I use it in my code with applied patch:

          private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) {
              Map<String, String> serdePropertiesMap = new HashMap<>();
      
              serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName());
              serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE);
              serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor));
      
      
              serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName());
              /* Use different key factory for simple and composite primary key */
              if (tableDescriptor.getPkDescriptors().size() == 1) {
                  serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName());
              } else {
                  serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName());
              }
      
              String serDeProperties = serdePropertiesMap.entrySet().stream()
                      .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue()))
                      .collect(Collectors.joining(COLUMNS_SEPARATOR));
      
              logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
              return serDeProperties;
          }
      

        Attachments

        1. HIVE-11233.1.patch
          70 kB
          Svetozar Ivanov
        2. HIVE-11233.2.patch
          69 kB
          Svetozar Ivanov
        3. HIVE-11233.3.patch
          69 kB
          Svetozar Ivanov
        4. HIVE-11233.4.patch
          69 kB
          Svetozar Ivanov
        5. HIVE-11233.5.patch
          69 kB
          Svetozar Ivanov
        6. HIVE-11233.patch
          70 kB
          Svetozar Ivanov
        7. HIVE-11233-branch-1.2.patch
          71 kB
          Svetozar Ivanov
        8. HIVE-11233-branch-2.0.patch
          70 kB
          Svetozar Ivanov

          Activity

            People

            • Assignee:
              svetozari Svetozar Ivanov
              Reporter:
              svetozari Svetozar Ivanov
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: