[PIG-953] Enable merge join in pig to work with loaders and store functions which can internally index sorted data

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.3.0
• Fix Version/s: 0.6.0
• Hadoop Flags: Reviewed
• Assignee: Pradeep Kamath
• Reporter: Pradeep Kamath
• Created: 10/Sep/09 18:47
• Updated: 24/Mar/10 22:15
• Resolved: 29/Oct/09 17:29

Description

Currently merge join implementation in pig includes construction of an index on sorted data and use of that index to seek into the "right input" to efficiently perform the join operation. Some loaders (notably the zebra loader) internally implement an index on sorted data and can perform this seek efficiently using their index. So the use of the index needs to be abstracted in such a way that when the loader supports indexing, pig uses it (indirectly through the loader) and does not construct an index.

Issue Links

• is blocked by: PIG-981 Merge join should restrict join key expressions to simple projects