[HIVE-4403] Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0, 0.11.0
• Fix Version/s: 0.12.0
• Assignee: Chu Tong
• Reporter: Mark Grover
• Created: 23/Apr/13 17:45
• Updated: 15/Oct/13 23:28
• Resolved: 03/Jun/13 14:39

Description

While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings related to overriding final parameters in job.conf. This was on a pseudo distributed cluster. FWIW, I didn't see this happen on a fully-distributed cluster. Perhaps, Hive's job.conf is overriding some final parameters it shouldn't.

Here is what the warnings looked like:

2013-04-19 14:20:32,304 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2013-04-19 14:20:32,367 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.

To reproduce, run a query like:

CREATE TABLE u_data (
  userid INT,
  movieid INT,
  rating INT,
  unixtime STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;

Load some data into u_data, here is some sample data:
https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data

Run a simple query on that data (on YARN/MR2)

INSERT OVERWRITE DIRECTORY '/tmp/count'
SELECT COUNT(1) FROM u_data

Issue Links

• is related to: BIGTOP-885 TestHiveSmokeBulk fails on Hive 0.9