Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13584

HBaseStorageHandler should support table pre-split

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 2.0.0
    • None
    • HBase Handler
    • Hide
      - This patch gives opportunity for creation of pre-splitted HBase table via Hive. Via TBLPROPERTIES could be configured following for HBase table:
      - Splits number
      - Split algorithm (if is not configured by default would be used 'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm)
      - Split algorithm first row key
      - Split algorithm last row key

      For more details see provided example in description field of this issue.
      Show
      - This patch gives opportunity for creation of pre-splitted HBase table via Hive. Via TBLPROPERTIES could be configured following for HBase table: - Splits number - Split algorithm (if is not configured by default would be used 'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm) - Split algorithm first row key - Split algorithm last row key For more details see provided example in description field of this issue.

    Description

      Currently is not possible creation of HBase table via HBaseStorageHandler with specified pre-split strategy.

      I use it in my code with applied patch in that way:

      public String buildCreateTblStatement(TableDescriptor tableDescriptor) {
              StringBuilder createTableStatement = ...;
      
              ...
              ...
              String tblProperties = buildTblProperties(tableDescriptor);
              if (StringUtils.isNotBlank(tblProperties)) {
                   createTableStatement.append(" TBLPROPERTIES (" + tblProperties + ")");
              }
               ...
              return createTableStatement.toString();
      }
      
      
      private static String buildTblProperties(TableDescriptor tableDescriptor) {
      
              ...
              ...
      
              if (supportsPreSplit(tableDescriptor)) {
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLITS_NUMBER));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(String.valueOf(tableDescriptor.getNumRegions())));
      
                  GeneratedPKDescriptor columnDescriptor = (GeneratedPKDescriptor) tableDescriptor.getPkDescriptors().get(0);
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_FIRSTROW_KEY));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().firstRowKey()));
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_LASTROW_KEY));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().lastRowKey()));
              }
      
              logger.trace("TBLPROPERTIES are [{}]", sb);
              return sb.toString();
      }
      

      Attachments

        1. HIVE-13584-branch-2.0.patch
          3 kB
          Svetozar Ivanov
        2. HIVE-13584.patch
          3 kB
          Svetozar Ivanov
        3. HIVE-13584.1.patch
          5 kB
          Svetozar Ivanov
        4. HIVE-13584.2.patch
          13 kB
          Svetozar Ivanov
        5. HIVE-13584.3.patch
          13 kB
          Svetozar Ivanov
        6. HIVE-13584.4.patch
          12 kB
          Svetozar Ivanov

        Activity

          People

            svetozari Svetozar Ivanov
            svetozari Svetozar Ivanov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: