Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20720

Add partition column option to JDBC handler



    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • StorageHandler
    • None
    • Reviewed


      Currently JdbcStorageHandler does not split input in Tez. The reason is numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. SQLStdAuth) is in use. User ends up always use 1 split.

      We need to rely on this new feature if we want to support multi-splits. Here is my proposal:
      1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in tblproperties if user want to split jdbc data source. In case lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min query to get this in planner. We can currently limit partitionColumn to only numeric/date/timestamp column for simplicity
      2. If partitionColumn/numPartitions are not specified, don't split input
      3. Splits are equal intervals without respect to data distribution
      4. There is also a "hive.sql.query.split" flag vetos the split (can be set manually or automatically by calcite)
      5. If partitionColumn is not defined, but numPartitions is defined, use original limit/offset logic (however, don't rely on numSplit).


        1. HIVE-20720.1.patch
          50 kB
          Daniel Dai
        2. HIVE-20720.10.patch
          114 kB
          Daniel Dai
        3. HIVE-20720.2.patch
          77 kB
          Daniel Dai
        4. HIVE-20720.3.patch
          78 kB
          Daniel Dai
        5. HIVE-20720.4.patch
          78 kB
          Daniel Dai
        6. HIVE-20720.5.patch
          109 kB
          Daniel Dai
        7. HIVE-20720.6.patch
          110 kB
          Daniel Dai
        8. HIVE-20720.7.patch
          112 kB
          Daniel Dai
        9. HIVE-20720.8.patch
          114 kB
          Daniel Dai
        10. HIVE-20720.9.patch
          114 kB
          Daniel Dai

        Issue Links



              daijy Daniel Dai
              daijy Daniel Dai
              0 Vote for this issue
              2 Start watching this issue