[HIVE-7155] WebHCat controller job exceeds container memory limit

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.14.0
• Component/s: WebHCat
• Assignee: shanyu zhao
• Reporter: shanyu zhao
• Created: 31/May/14 01:28
• Updated: 09/Apr/15 05:26
• Resolved: 09/Jun/14 20:35

Description

Submit a Hive query on a large table via WebHCat results in failure because the WebHCat controller job is killed by Yarn since it exceeds the memory limit (set by mapreduce.map.memory.mb, defaults to 1GB):

 INSERT OVERWRITE TABLE Temp_InjusticeEvents_2014_03_01_00_00 SELECT * from Stage_InjusticeEvents where LogTimestamp > '2014-03-01 00:00:00' and LogTimestamp <= '2014-03-01 01:00:00';

We could increase mapreduce.map.memory.mb to solve this problem, but this way we are changing this setting system wise.

We need to provide a WebHCat configuration to overwrite mapreduce.map.memory.mb when submitting the controller job.

Issue Links

• relates to: HIVE-7948 Add an E2E test to verify fix for HIVE-7155
• links to: review board