[HIVE-4006] Add local sort operator

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Query Processor
• Assignee: Namit Jain
• Reporter: Namit Jain
• Created: 11/Feb/13 09:55
• Updated: 02/May/13 02:30

Description

We've seen in the past that sorting data on a specific column can greatly improve the compression of data. The problem is that sorting data is expensive and requires a reduce phase.

One way around this is to add a local sort (either as an operator or between serialization and output). This could take chunks of rows and do an in memory sort of these. This would be much faster, but would need to be very memory efficient in order to get the maximum number of rows in a chunk (and hence the maximum benefit).

Issue Links

• depends upon: HIVE-4007 Create abstract classes for serializer and deserializer