Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13584

HBaseStorageHandler should support table pre-split

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: HBase Handler
    • Labels:
    • Target Version/s:
    • Release Note:
      Hide
      - This patch gives opportunity for creation of pre-splitted HBase table via Hive. Via TBLPROPERTIES could be configured following for HBase table:
      - Splits number
      - Split algorithm (if is not configured by default would be used 'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm)
      - Split algorithm first row key
      - Split algorithm last row key

      For more details see provided example in description field of this issue.
      Show
      - This patch gives opportunity for creation of pre-splitted HBase table via Hive. Via TBLPROPERTIES could be configured following for HBase table: - Splits number - Split algorithm (if is not configured by default would be used 'org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit' algorithm) - Split algorithm first row key - Split algorithm last row key For more details see provided example in description field of this issue.

      Description

      Currently is not possible creation of HBase table via HBaseStorageHandler with specified pre-split strategy.

      I use it in my code with applied patch in that way:

      public String buildCreateTblStatement(TableDescriptor tableDescriptor) {
              StringBuilder createTableStatement = ...;
      
              ...
              ...
              String tblProperties = buildTblProperties(tableDescriptor);
              if (StringUtils.isNotBlank(tblProperties)) {
                   createTableStatement.append(" TBLPROPERTIES (" + tblProperties + ")");
              }
               ...
              return createTableStatement.toString();
      }
      
      
      private static String buildTblProperties(TableDescriptor tableDescriptor) {
      
              ...
              ...
      
              if (supportsPreSplit(tableDescriptor)) {
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLITS_NUMBER));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(String.valueOf(tableDescriptor.getNumRegions())));
      
                  GeneratedPKDescriptor columnDescriptor = (GeneratedPKDescriptor) tableDescriptor.getPkDescriptors().get(0);
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_FIRSTROW_KEY));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().firstRowKey()));
                  sb.append(", ");
                  sb.append(quoteInSingleQuotes(HBaseStorageHandler.SPLIT_ALGORITHM_LASTROW_KEY));
                  sb.append(" = ");
                  sb.append(quoteInSingleQuotes(columnDescriptor.getPkGenerator().lastRowKey()));
              }
      
              logger.trace("TBLPROPERTIES are [{}]", sb);
              return sb.toString();
      }
      

        Attachments

        1. HIVE-13584.4.patch
          12 kB
          Svetozar Ivanov
        2. HIVE-13584.3.patch
          13 kB
          Svetozar Ivanov
        3. HIVE-13584.2.patch
          13 kB
          Svetozar Ivanov
        4. HIVE-13584.1.patch
          5 kB
          Svetozar Ivanov
        5. HIVE-13584.patch
          3 kB
          Svetozar Ivanov
        6. HIVE-13584-branch-2.0.patch
          3 kB
          Svetozar Ivanov

          Activity

            People

            • Assignee:
              svetozari Svetozar Ivanov
              Reporter:
              svetozari Svetozar Ivanov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: