[CALCITE-853] EnumerableAggregate should take advantage of input collation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.23.0
• Assignee: Unassigned
• Reporter: liyang
• Created: 21/Aug/15 22:58
• Updated: 29/May/21 00:43
• Resolved: 29/May/21 00:43

Description

Li Yang <[e-mail]>
Aug 20 (2 days ago)

I encountered Out Of Mem exception when a huge result set is passed into EnumerableAggregate and get aggregated in memory. I'm thinking if the input is sorted by the group-by key, then the groupBy() don't have to hold all data in memory any more.

Julian Hyde <[e-mail]>
2:20 PM (16 hours ago)

Yes, that would be useful. Please log a jira.

Enumerable.groupBy doesn't know its input's collation so can't make that decision, but EnumerableAggregate does. I think that EnumerableAggregate should have a "trigger key", a subset of its group key, and if the trigger key changes it will emit and flush its hash table.

As well as for your use case, it will be useful for streaming queries.

Issue Links

• is related to: CALCITE-2540 Streaming Sort relational operator
• relates to: CALCITE-784 LogicalAggregate's create method discards any collation traits from input