Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4963

Support in memory PTF partitions

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: PTF-Windowing
    • Labels:

      Description

      PTF partitions apply the defensive mode of assuming that partitions will not fit in memory. Because of this there is a significant deserialization overhead when accessing elements.

      Allow the user to specify that there is enough memory to hold partitions through a 'hive.ptf.partition.fits.in.mem' option.

      Savings depends on partition size and in case of windowing the number of UDAFs and the window ranges. For eg for the following (admittedly extreme) case the PTFOperator exec times went from 39 secs to 8 secs.

      select t, s, i, b, f, d,
      min(t) over(partition by 1 rows between unbounded preceding and current row), 
      min(s) over(partition by 1 rows between unbounded preceding and current row), 
      min(i) over(partition by 1 rows between unbounded preceding and current row), 
      min(b) over(partition by 1 rows between unbounded preceding and current row) 
      from over10k
      

        Attachments

        1. PTFRowContainer.patch
          19 kB
          Harish Butani
        2. HIVE-4963.D12279.3.patch
          80 kB
          Phabricator
        3. HIVE-4963.D12279.2.patch
          68 kB
          Phabricator
        4. HIVE-4963.D12279.1.patch
          68 kB
          Phabricator
        5. HIVE-4963.D11955.1.patch
          28 kB
          Phabricator

          Issue Links

            Activity

              People

              • Assignee:
                rhbutani Harish Butani
                Reporter:
                rhbutani Harish Butani
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: