With this patch, it is now possible to perform map-side cogroup if data is sorted and loader implements certain interfaces. Primary algorithm is based on sort-merge join with additional restrictions.
Following preconditions must be met to use this feature:
1) No other operations can be done between load and cogroup statements.
2) Data must be sorted on join keys for all tables in ASC order.
3) Nulls are considered smaller then everything. So, if data contains null keys, they should occur before anything else.
4) Left-most loader must implement {CollectableLoader} interface as well as {OrderedLoadFunc}.
5) All other loaders must implement IndexableLoadFunc.
6) Type information must be provided in schema for all the loaders.
Note that Zebra loader satisfies all of these conditions, so can be used out of box.
Similar conditions apply to map-side outer joins (using merge) (
PIG-1353) as well.
Example:
A = load 'data1' using org.apache.hadoop.zebra.pig.TableLoader('id:int', 'sorted');
B = load 'data2' using org.apache.hadoop.zebra.pig.TableLoader('id:int', 'sorted');
C = COGROUP A by id, B by id using 'merge';