Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20720

Add partition column option to JDBC handler

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • StorageHandler
    • None
    • Reviewed

    Description

      Currently JdbcStorageHandler does not split input in Tez. The reason is numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. SQLStdAuth) is in use. User ends up always use 1 split.

      We need to rely on this new feature if we want to support multi-splits. Here is my proposal:
      1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound in tblproperties if user want to split jdbc data source. In case lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min query to get this in planner. We can currently limit partitionColumn to only numeric/date/timestamp column for simplicity
      2. If partitionColumn/numPartitions are not specified, don't split input
      3. Splits are equal intervals without respect to data distribution
      4. There is also a "hive.sql.query.split" flag vetos the split (can be set manually or automatically by calcite)
      5. If partitionColumn is not defined, but numPartitions is defined, use original limit/offset logic (however, don't rely on numSplit).

      Attachments

        1. HIVE-20720.1.patch
          50 kB
          Daniel Dai
        2. HIVE-20720.10.patch
          114 kB
          Daniel Dai
        3. HIVE-20720.2.patch
          77 kB
          Daniel Dai
        4. HIVE-20720.3.patch
          78 kB
          Daniel Dai
        5. HIVE-20720.4.patch
          78 kB
          Daniel Dai
        6. HIVE-20720.5.patch
          109 kB
          Daniel Dai
        7. HIVE-20720.6.patch
          110 kB
          Daniel Dai
        8. HIVE-20720.7.patch
          112 kB
          Daniel Dai
        9. HIVE-20720.8.patch
          114 kB
          Daniel Dai
        10. HIVE-20720.9.patch
          114 kB
          Daniel Dai

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            daijy Daniel Dai Assign to me
            daijy Daniel Dai
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment