[CRUNCH-118] Add Aggregator and lib functionality for computing the distinct elements of a PCollection

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: MapReduce Patterns
• Assignee: Josh Wills
• Reporter: Josh Wills
• Created: 21/Nov/12 17:31
• Updated: 22/Feb/13 06:57
• Resolved: 25/Nov/12 09:21

Description

In my continuing effort to add functions that I like from Hive/Pig to Crunch, I'm proposing a) an Aggregator that behaves like Hive's collect_set() UDAF and b) an o.a.c.lib.Distinct library for computing the distinct elements of a PCollection via a groupByKey operation.