[HIVE-4078] Delay the serialize-deserialize pair in CommonJoinTaskDispatcher

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: Query Processor
• Labels: client perfomance
• Release Note: Create a copy of the MapredWork only if a conditional task is involved and avoid the copy if the task is non-conditional.
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 27/Feb/13 01:53
• Updated: 03/Feb/18 20:21
• Resolved: 11/Apr/13 04:55

Description

CommonJoinProcessor tries to clone a MapredWork while attempting a conversion to a map-join

  // deep copy a new mapred work from xml
  InputStream in = new ByteArrayInputStream(xml.getBytes("UTF-8"));
  MapredWork newWork = Utilities.deserializeMapRedWork(in, physicalContext.getConf());

which is a very heavy operation memory wise & cpu-wise.

It would be better to do this only if a conditional task is required, resulting in a copy of the task.

Issue Links

• is blocked by: HIVE-2340 optimize orderby followed by a groupby
• is blocked by: HIVE-3891 physical optimizer changes for auto sort-merge join
• is blocked by: HIVE-4071 Map-join outer join produces incorrect results.
• is blocked by: HIVE-3952 merge map-job followed by map-reduce job
• is blocked by: HIVE-3996 Correctly enforce the memory limit on the multi-table map-join