[PIG-686] PERFORMANCE: improve how data is stored between M-R jobs and between Map and Reduce

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.2.0
• Assignee: Unassigned
• Reporter: Olga Natkovich
• Created: 26/Feb/09 19:56
• Updated: 12/Jun/09 00:38
• Resolved: 12/Jun/09 00:38

Description

Currently, there is quite a bit of overhead in how the data is serialized in both cases because a type information is stored with each field.

However, most of the time the data has known and consistent schema in which case, it is sufficient to store the schema once.

This change could really decrease the ammount of intermediate data generated.