Hive
  1. Hive
  2. HIVE-4403

Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings related to overriding final parameters in job.conf. This was on a pseudo distributed cluster. FWIW, I didn't see this happen on a fully-distributed cluster. Perhaps, Hive's job.conf is overriding some final parameters it shouldn't.

      Here is what the warnings looked like:

      2013-04-19 14:20:32,304 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
      2013-04-19 14:20:32,367 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
      

      To reproduce, run a query like:

      CREATE TABLE u_data (
        userid INT,
        movieid INT,
        rating INT,
        unixtime STRING)
      ROW FORMAT DELIMITED
      FIELDS TERMINATED BY '\t'
      STORED AS TEXTFILE;
      

      Load some data into u_data, here is some sample data:
      https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data

      Run a simple query on that data (on YARN/MR2)

      INSERT OVERWRITE DIRECTORY '/tmp/count'
      SELECT COUNT(1) FROM u_data
      
      1. HIVE-4403.patch
        1 kB
        Chu Tong
      2. HIVE-4403.patch
        1 kB
        Chu Tong

        Issue Links

          Activity

          Mark Grover created issue -
          Mark Grover made changes -
          Field Original Value New Value
          Link This issue is related too BIGTOP-885 [ BIGTOP-885 ]
          Mark Grover made changes -
          Affects Version/s 0.10.0 [ 12320745 ]
          Mark Grover made changes -
          Description While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings related to overriding final parameters in job.conf. Perhaps, Hive's job.conf is overriding some final parameters it shouldn't.

          Here is what the warnings looked like:
          {code}
          2013-04-19 14:20:32,304 WARN [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
          2013-04-19 14:20:32,367 WARN [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
          {code}

          To reproduce, run a query like:
          {code}
          CREATE TABLE u_data (
            userid INT,
            movieid INT,
            rating INT,
            unixtime STRING)
          ROW FORMAT DELIMITED
          FIELDS TERMINATED BY '\t'
          STORED AS TEXTFILE;
          {code}
          Load some data into u_data, here is some sample data:
          https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data

          Run a simple query on that data (on YARN/MR2)
          {code}
          INSERT OVERWRITE DIRECTORY '/tmp/count'
          SELECT COUNT(1) FROM u_data
          {code}
          While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings related to overriding final parameters in job.conf. This was on a pseudo distributed cluster. FWIW, I didn't see this happen on a fully-distributed cluster. Perhaps, Hive's job.conf is overriding some final parameters it shouldn't.

          Here is what the warnings looked like:
          {code}
          2013-04-19 14:20:32,304 WARN [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
          2013-04-19 14:20:32,367 WARN [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
          {code}

          To reproduce, run a query like:
          {code}
          CREATE TABLE u_data (
            userid INT,
            movieid INT,
            rating INT,
            unixtime STRING)
          ROW FORMAT DELIMITED
          FIELDS TERMINATED BY '\t'
          STORED AS TEXTFILE;
          {code}
          Load some data into u_data, here is some sample data:
          https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data

          Run a simple query on that data (on YARN/MR2)
          {code}
          INSERT OVERWRITE DIRECTORY '/tmp/count'
          SELECT COUNT(1) FROM u_data
          {code}
          Gavin made changes -
          Link This issue is related to BIGTOP-885 [ BIGTOP-885 ]
          Gavin made changes -
          Link This issue is related to BIGTOP-885 [ BIGTOP-885 ]
          Chu Tong made changes -
          Attachment HIVE-4403.patch [ 12582212 ]
          Chu Tong made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Assignee Chu Tong [ stayhf ]
          Chu Tong made changes -
          Attachment HIVE-4403.patch [ 12585779 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.12.0 [ 12324312 ]
          Resolution Fixed [ 1 ]
          Ashutosh Chauhan made changes -
          Affects Version/s 0.11.0 [ 12323587 ]
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Chu Tong
              Reporter:
              Mark Grover
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development