[PIG-4652] [Pig on Tez] Key Comparison is slower than mapreduce

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.18.0
• Assignee: Unassigned
• Reporter: Rohini Palaniswamy
• Created: 07/Aug/15 22:10
• Updated: 24/May/17 23:03

Description

Tez is using PigTupleSortComparator on both map and reduce side and in POShuffleTezLoad. Mapreduce is using PigTupleWritableComparator on the map and reduce side for comparing tuples which is byte only comparison and very fast. It then uses PigGrouping<DataType>WritableComparator as the grouping comparator to correctly group those keys.

It is not possible to use similar method in Tez (PigTupleWritableComparator for output and input and PigTupleSortComparator in POShuffleTezLoad), without addition of APIs in Tez to get raw bytes of the keys. Because when we compare multiple inputs for min key in POShuffleTezLoad, there raw bytes need to be compared to maintain the same order as the mapside. In mapreduce, there was only single input and mapreduce framework sorted them together. But in Tez, the join inputs are sorted separately and the application only gets the serialized key. Need APIs in Tez KeyValuesReader to get the bytes of the current key as well which can be used in POShuffleTezLoad for min key comparison.

Issue Links

• requires: TEZ-2715 Add API to Reader classes to expose raw key bytes in addition to getCurrentKey

Sub-Tasks

• [Pig on Tez] Optimize GroupBy and Distinct key comparison: Closed
• Reduce key comparisons in TezAccumulativeTupleBuffer: Open