Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13584

HBaseStorageHandler should support table pre-split

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 2.0.0
    • None
    • HBase Handler
    • Hide
      - This patch gives opportunity for creation of pre-splitted HBase table via Hive. Via TBLPROPERTIES could be configured following for HBase table:
      - Splits number
      - Split algorithm (if is not configured by default would be used 'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm)
      - Split algorithm first row key
      - Split algorithm last row key

      For more details see provided example in description field of this issue.
      Show
      - This patch gives opportunity for creation of pre-splitted HBase table via Hive. Via TBLPROPERTIES could be configured following for HBase table: - Splits number - Split algorithm (if is not configured by default would be used 'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm) - Split algorithm first row key - Split algorithm last row key For more details see provided example in description field of this issue.

    Description

      Currently is not possible creation of HBase table via HBaseStorageHandler with specified pre-split strategy.

      I use it in my code with applied patch in that way:

      public String buildCreateTblStatement(TableDescriptor tableDescriptor) {
              StringBuilder createTableStatement = ...;
      
              ...
              ...
              String tblProperties = buildTblProperties(tableDescriptor);
              if (StringUtils.isNotBlank(tblProperties)) {
                   createTableStatement.append(" TBLPROPERTIES (" + tblProperties + ")");
              }
               ...
              return createTableStatement.toString();
      }
      
      
      private static String buildTblProperties(TableDescriptor tableDescriptor) {
      
              ...
              ...
      
              if (supportsPreSplit(tableDescriptor)) {
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLITS_NUMBER));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(String.valueOf(tableDescriptor.getNumRegions())));
      
                  GeneratedPKDescriptor columnDescriptor = (GeneratedPKDescriptor) tableDescriptor.getPkDescriptors().get(0);
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_FIRSTROW_KEY));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().firstRowKey()));
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_LASTROW_KEY));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().lastRowKey()));
              }
      
              logger.trace("TBLPROPERTIES are [{}]", sb);
              return sb.toString();
      }
      

      Attachments

        1. HIVE-13584.1.patch
          5 kB
          Svetozar Ivanov
        2. HIVE-13584.2.patch
          13 kB
          Svetozar Ivanov
        3. HIVE-13584.3.patch
          13 kB
          Svetozar Ivanov
        4. HIVE-13584.4.patch
          12 kB
          Svetozar Ivanov
        5. HIVE-13584.patch
          3 kB
          Svetozar Ivanov
        6. HIVE-13584-branch-2.0.patch
          3 kB
          Svetozar Ivanov

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            svetozari Svetozar Ivanov Assign to me
            svetozari Svetozar Ivanov

            Dates

              Created:
              Updated:

              Slack

                Issue deployment