Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-865

Add feature that enables PK-chunking in partition

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      In SFDC(salesforce) connector, we have partitioning mechanisms to split a giant query to multiple sub queries. There are 3 mechanisms:

      • simple partition (equally split by time)
      • dynamic pre-partition (generate histogram and split by row numbers)
      • user specified partition (set up time range in job file)

      However there are tables like Task and Contract are failing time to time to fetch full data.

      We may want to utilize PK-chunking to partition the query.

       

      The pk-chunking doc from SFDC - https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                arekusuri Alex Li
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 9h 20m
                  9h 20m