[HIVE-6262] Remove unnecessary copies of schema + table desc from serialized plan

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Gunther Hagleitner
• Reporter: Gunther Hagleitner
• Created: 22/Jan/14 07:50
• Updated: 29/Jan/14 06:51
• Resolved: 29/Jan/14 06:51

Description

Currently for a partitioned table the following are true:

• for each partitiondesc we send a copy of the corresponding tabledesc
• for each partitiondesc we send two copies of the schema (in different formats).

Obviously we need to send different schemas if they are required by schema evolution, but in our case we'll always end up with multiple copies.

The effect can be dramatic. The reductions by removing those on partitioned tables easily be can be 8-10x in size. Plans themselves can be 10s to 100s of mb (even with kryo). The size difference also plays out in every task on the cluster we run.