Hive
  1. Hive
  2. HIVE-4403

Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings related to overriding final parameters in job.conf. This was on a pseudo distributed cluster. FWIW, I didn't see this happen on a fully-distributed cluster. Perhaps, Hive's job.conf is overriding some final parameters it shouldn't.

      Here is what the warnings looked like:

      2013-04-19 14:20:32,304 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
      2013-04-19 14:20:32,367 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
      

      To reproduce, run a query like:

      CREATE TABLE u_data (
        userid INT,
        movieid INT,
        rating INT,
        unixtime STRING)
      ROW FORMAT DELIMITED
      FIELDS TERMINATED BY '\t'
      STORED AS TEXTFILE;
      

      Load some data into u_data, here is some sample data:
      https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data

      Run a simple query on that data (on YARN/MR2)

      INSERT OVERWRITE DIRECTORY '/tmp/count'
      SELECT COUNT(1) FROM u_data
      
      1. HIVE-4403.patch
        1 kB
        Chu Tong
      2. HIVE-4403.patch
        1 kB
        Chu Tong

        Issue Links

          Activity

          Mark Grover created issue -
          Mark Grover made changes -
          Field Original Value New Value
          Link This issue is related too BIGTOP-885 [ BIGTOP-885 ]
          Mark Grover made changes -
          Affects Version/s 0.10.0 [ 12320745 ]
          Mark Grover made changes -
          Description While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings related to overriding final parameters in job.conf. Perhaps, Hive's job.conf is overriding some final parameters it shouldn't.

          Here is what the warnings looked like:
          {code}
          2013-04-19 14:20:32,304 WARN [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
          2013-04-19 14:20:32,367 WARN [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
          {code}

          To reproduce, run a query like:
          {code}
          CREATE TABLE u_data (
            userid INT,
            movieid INT,
            rating INT,
            unixtime STRING)
          ROW FORMAT DELIMITED
          FIELDS TERMINATED BY '\t'
          STORED AS TEXTFILE;
          {code}
          Load some data into u_data, here is some sample data:
          https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data

          Run a simple query on that data (on YARN/MR2)
          {code}
          INSERT OVERWRITE DIRECTORY '/tmp/count'
          SELECT COUNT(1) FROM u_data
          {code}
          While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings related to overriding final parameters in job.conf. This was on a pseudo distributed cluster. FWIW, I didn't see this happen on a fully-distributed cluster. Perhaps, Hive's job.conf is overriding some final parameters it shouldn't.

          Here is what the warnings looked like:
          {code}
          2013-04-19 14:20:32,304 WARN [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
          2013-04-19 14:20:32,367 WARN [main] conf.Configuration (Configuration.java:loadProperty(2032)) - file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
          {code}

          To reproduce, run a query like:
          {code}
          CREATE TABLE u_data (
            userid INT,
            movieid INT,
            rating INT,
            unixtime STRING)
          ROW FORMAT DELIMITED
          FIELDS TERMINATED BY '\t'
          STORED AS TEXTFILE;
          {code}
          Load some data into u_data, here is some sample data:
          https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data

          Run a simple query on that data (on YARN/MR2)
          {code}
          INSERT OVERWRITE DIRECTORY '/tmp/count'
          SELECT COUNT(1) FROM u_data
          {code}
          Gavin made changes -
          Link This issue is related to BIGTOP-885 [ BIGTOP-885 ]
          Gavin made changes -
          Link This issue is related to BIGTOP-885 [ BIGTOP-885 ]
          Chu Tong made changes -
          Attachment HIVE-4403.patch [ 12582212 ]
          Hide
          Chu Tong added a comment -

          The reason for the warnings presented in this bug is Hive client tries to override some Hadoop final parameters defined in mapred-default.xml. To dig it deeper, Hive configuration inherits Hadoop configuration and the current way Hive overrides Hadoop parameters is:

          1) Create a default Hadoop configuration (this contains all default Hadoop parameters including the ones defined as final).
          2) Overlay parameters it wants to override on configuration created in 1).
          3) Overlay configuration generated in 2) over the default Hadoop parameters it inherits (these default Hadoop parameters contains all default Hadoop parameters including the ones defined as final again).

          Since configuration generated in 2) contains all the default Hadoop parameters and when it comes to 3), the warning is thrown.

          Solution to resolve this problem is when 1) happens, instead of create a default Hadoop configuration, an empty Hadoop configuration is created and 2) overlays Hive parameters on this empty configuration. This way, in 3), configuration in 2) will override any default Hadoop parameters it wants to overrides, however, no warning will be thrown as 2) does not contain default Hadoop parameters.

          I have tested this by different code path including:

          1) keep everything as default
          2) define overriding parameters in hive-site.xml
          3) define overriding parameters in hive client shell

          and all these cases work well.

          Show
          Chu Tong added a comment - The reason for the warnings presented in this bug is Hive client tries to override some Hadoop final parameters defined in mapred-default.xml. To dig it deeper, Hive configuration inherits Hadoop configuration and the current way Hive overrides Hadoop parameters is: 1) Create a default Hadoop configuration (this contains all default Hadoop parameters including the ones defined as final). 2) Overlay parameters it wants to override on configuration created in 1). 3) Overlay configuration generated in 2) over the default Hadoop parameters it inherits (these default Hadoop parameters contains all default Hadoop parameters including the ones defined as final again). Since configuration generated in 2) contains all the default Hadoop parameters and when it comes to 3), the warning is thrown. Solution to resolve this problem is when 1) happens, instead of create a default Hadoop configuration, an empty Hadoop configuration is created and 2) overlays Hive parameters on this empty configuration. This way, in 3), configuration in 2) will override any default Hadoop parameters it wants to overrides, however, no warning will be thrown as 2) does not contain default Hadoop parameters. I have tested this by different code path including: 1) keep everything as default 2) define overriding parameters in hive-site.xml 3) define overriding parameters in hive client shell and all these cases work well.
          Chu Tong made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Chu Tong added a comment -

          Can someone please help me to review this code? Thanks

          Show
          Chu Tong added a comment - Can someone please help me to review this code? Thanks
          Hide
          Mark Grover added a comment -

          Chu, sorry about the delay. I can test patched Hive to see if this resolves my original problem, please give me a few days.

          In the meanwhile, even though this is a small patch, do you mind uploading it on Reviewboard or Phabricator? Thanks for contributing!

          Show
          Mark Grover added a comment - Chu, sorry about the delay. I can test patched Hive to see if this resolves my original problem, please give me a few days. In the meanwhile, even though this is a small patch, do you mind uploading it on Reviewboard or Phabricator? Thanks for contributing!
          Hide
          Ashutosh Chauhan added a comment -

          Patch doesn't apply cleanly on trunk anymore. Chu Tong Can you rebase it on trunk ?

          Show
          Ashutosh Chauhan added a comment - Patch doesn't apply cleanly on trunk anymore. Chu Tong Can you rebase it on trunk ?
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Assignee Chu Tong [ stayhf ]
          Hide
          Chu Tong added a comment -

          Sync the changes with trunk

          Show
          Chu Tong added a comment - Sync the changes with trunk
          Chu Tong made changes -
          Attachment HIVE-4403.patch [ 12585779 ]
          Hide
          Chu Tong added a comment -

          Mark Grover, I tried to search on the Facebook Phabricator and I can't find this bug there. Do I need to create on there first and upload the patch to it afterwards? Thanks a lot.

          Show
          Chu Tong added a comment - Mark Grover , I tried to search on the Facebook Phabricator and I can't find this bug there. Do I need to create on there first and upload the patch to it afterwards? Thanks a lot.
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Mark Grover added a comment -

          Ashutosh +1'ed it. So, I guess there is no need for Phabricator now

          Thanks Chu and Ashutosh!

          Show
          Mark Grover added a comment - Ashutosh +1'ed it. So, I guess there is no need for Phabricator now Thanks Chu and Ashutosh!
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Chu!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Chu!
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.12.0 [ 12324312 ]
          Resolution Fixed [ 1 ]
          Ashutosh Chauhan made changes -
          Affects Version/s 0.11.0 [ 12323587 ]
          Hide
          Chu Tong added a comment -

          no problem, thank you for reviewing it Ashutosh

          Show
          Chu Tong added a comment - no problem, thank you for reviewing it Ashutosh
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #2126 (See https://builds.apache.org/job/Hive-trunk-h0.21/2126/)
          HIVE-4403 : Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters (Chu Tong via Ashutosh Chauhan) (Revision 1489008)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489008
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #2126 (See https://builds.apache.org/job/Hive-trunk-h0.21/2126/ ) HIVE-4403 : Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters (Chu Tong via Ashutosh Chauhan) (Revision 1489008) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489008 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          Hide
          Chu Tong added a comment -

          I took a look at the Hudson build log and seems it is because of a build failure instead of test failure due to newly added functionality/bug fix.

          Show
          Chu Tong added a comment - I took a look at the Hudson build log and seems it is because of a build failure instead of test failure due to newly added functionality/bug fix.
          Hide
          Mark Grover added a comment -

          Chu, that's fine. Nothing required from your side.

          Show
          Mark Grover added a comment - Chu, that's fine. Nothing required from your side.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #224 (See https://builds.apache.org/job/Hive-trunk-hadoop2/224/)
          HIVE-4403 : Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters (Chu Tong via Ashutosh Chauhan) (Revision 1489008)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489008
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #224 (See https://builds.apache.org/job/Hive-trunk-hadoop2/224/ ) HIVE-4403 : Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters (Chu Tong via Ashutosh Chauhan) (Revision 1489008) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489008 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          14d 6h 28m 1 Chu Tong 08/May/13 01:14
          Patch Available Patch Available Open Open
          25d 17h 29m 1 Ashutosh Chauhan 02/Jun/13 18:43
          Open Open Resolved Resolved
          20h 55m 1 Ashutosh Chauhan 03/Jun/13 15:39
          Resolved Resolved Closed Closed
          134d 8h 49m 1 Ashutosh Chauhan 16/Oct/13 00:28

            People

            • Assignee:
              Chu Tong
              Reporter:
              Mark Grover
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development