Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      Create a new operator that caches a number of record batches and then coordinates across the cluster on the distribution of partitioning keys to try to determine a reasonable set of range partitions. The outgoing stream should include a partition key that is equal to the width of the receiving fragment.

      • histogram or similar should be held in the distributed cache
      • need to figure out the logic for how long to wait before the partitioning estimate is good enough.
      • need to update the partitioning sender so that we can drop the partitioning column rather than sending it onward.
      1. DRILL-230_2013-10-23_13:10:50.patch
        206 kB
        Steven Phillips
      2. DRILL-230_2013-10-25_05:15:16.patch
        226 kB
        Steven Phillips
      3. DRILL-230_2013-10-25_05:16:58.patch
        218 kB
        Steven Phillips
      4. DRILL-230.patch
        120 kB
        Steven Phillips

        Activity

        Hide
        Steven Phillips added a comment -
        Show
        Steven Phillips added a comment - Created reviewboard https://reviews.apache.org/r/14764/
        Hide
        Steven Phillips added a comment -
        Show
        Steven Phillips added a comment - Updated reviewboard https://reviews.apache.org/r/14764/
        Hide
        Ashish Paliwal added a comment -

        Steven Phillips Jacques Nadeau Can the review be kept open for 2-3 more days? I have worked extensively on Caching and would like to review in detail. Have few more inputs but would like to read the code before adding them to review board.

        Steven Phillips We can get on a hangout to discuss further details. Let me know if it works.

        Show
        Ashish Paliwal added a comment - Steven Phillips Jacques Nadeau Can the review be kept open for 2-3 more days? I have worked extensively on Caching and would like to review in detail. Have few more inputs but would like to read the code before adding them to review board. Steven Phillips We can get on a hangout to discuss further details. Let me know if it works.
        Hide
        Steven Phillips added a comment -
        Show
        Steven Phillips added a comment - Updated reviewboard https://reviews.apache.org/r/14764/
        Hide
        Steven Phillips added a comment -
        Show
        Steven Phillips added a comment - Updated reviewboard https://reviews.apache.org/r/14764/
        Hide
        Jacques Nadeau added a comment -

        Merged in fe94aa8

        Show
        Jacques Nadeau added a comment - Merged in fe94aa8

          People

          • Assignee:
            Steven Phillips
            Reporter:
            Jacques Nadeau
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development