Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11233

Include Apache Phoenix support in HBaseStorageHandler

Log workAgile BoardRank to TopRank to BottomAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Release Note:
      Hide
      Thanks of this patch are possible read or write operations for HBase table in Apache Phoenix persistence mode. Following classes could be considered as 3 keypoints:
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory
      - org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory

      Note: See the code sample provided in description field of that issue for more details.
      Show
      Thanks of this patch are possible read or write operations for HBase table in Apache Phoenix persistence mode. Following classes could be considered as 3 keypoints: - org.apache.hadoop.hive.hbase.phoenix.PhoenixValueFactory - org.apache.hadoop.hive.hbase.phoenix.PhoenixKeyFactory - org.apache.hadoop.hive.hbase.phoenix.PhoenixCompositeKeyFactory Note: See the code sample provided in description field of that issue for more details.
    • Flags:
      Patch

      Description

      Currently HBaseStorageHandler doesn't provide mechanism for storage of binary sortable key and values. It is necessary when given HBase table is used for persistence by Apache Hive and Apache Phoenix. In that way all byte arrays read or written by Hive will be compatible with binary sortable format used in Phoenix.

      It turns out the major difference is in all numeric data types accordingly officially provided documentation - https://phoenix.apache.org/language/datatypes.html.

      That's how I use it in my code with applied patch:

          private static String buildWithSerDeProperties(TableDescriptor tableDescriptor) {
              Map<String, String> serdePropertiesMap = new HashMap<>();
      
              serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, tableDescriptor.getTableName());
              serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, BINARY_STORAGE_TYPE);
              serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, buildHBaseColumnsDefinition(tableDescriptor));
      
      
              serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, PhoenixValueFactory.class.getName());
              /* Use different key factory for simple and composite primary key */
              if (tableDescriptor.getPkDescriptors().size() == 1) {
                  serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, PhoenixKeyFactory.class.getName());
              } else {
                  serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, PhoenixCompositeKeyFactory.class.getName());
              }
      
              String serDeProperties = serdePropertiesMap.entrySet().stream()
                      .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + quoteInSingleQuotes(e.getValue()))
                      .collect(Collectors.joining(COLUMNS_SEPARATOR));
      
              logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
              return serDeProperties;
          }
      

        Attachments

        1. HIVE-11233.1.patch
          70 kB
          Svetozar Ivanov
        2. HIVE-11233.2.patch
          69 kB
          Svetozar Ivanov
        3. HIVE-11233.3.patch
          69 kB
          Svetozar Ivanov
        4. HIVE-11233.4.patch
          69 kB
          Svetozar Ivanov
        5. HIVE-11233.5.patch
          69 kB
          Svetozar Ivanov
        6. HIVE-11233.patch
          70 kB
          Svetozar Ivanov
        7. HIVE-11233-branch-1.2.patch
          71 kB
          Svetozar Ivanov
        8. HIVE-11233-branch-2.0.patch
          70 kB
          Svetozar Ivanov

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              svetozari Svetozar Ivanov Assign to me
              Reporter:
              svetozari Svetozar Ivanov

              Dates

              • Created:
                Updated:

                Issue deployment