[DRILL-3462] There appears to be no way to have complex intermediate state

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Ted Dunning
• Created: 06/Jul/15 20:54
• Updated: 06/Jul/15 20:54

Description

After spending several frustrating days on the problem (see also DRILL-3461), it appears that there is no viable idiom for building an aggregator that has internal state that is anything more than a scalar.

What is needed is:

1) The ability to allocate a Repeated* type for use in a Workspace variables. Currently, new works to get the basic structure, but there is no good way to allocate the corresponding vector.

2) The ability to use and to allocate a ComplexWriter in the Workspace variables.

3) The ability to write a UDAF that supports multi-phase aggregation. It would be just fine if I simply have to write a combine method on my UDAF class. I don't think that there is any way to infer such a combiner from the parameters and workspace variables. An alternative API would be to have a form of the output function that is given an Iterable<OutputClass>, but that is probably much less efficient than simply having a combine method that is called repeatedly.