Hive
  1. Hive
  2. HIVE-2789

query_properties.q contains non-deterministic queries

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Tests
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      query_properties.q test failure:

      [junit] Begin query: query_properties.q
      [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 17:00:27 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 17:00:27 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] 12/01/23 17:00:31 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
      [junit] 12/01/23 17:00:31 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
      [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]_[0-9] -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/query_properties.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/query_properties.q.out
      [junit] 91c91
      [junit] < 97 val_97
      [junit]
      [junit] > 238 val_238
      [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1
      [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.
      [junit] at junit.framework.Assert.fail(Assert.java:50)
      [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_query_properties(TestCliDriver.java:227)
      [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      [junit] at java.lang.reflect.Method.invoke(Method.java:616)
      [junit] at junit.framework.TestCase.runTest(TestCase.java:168)
      [junit] at junit.framework.TestCase.runBare(TestCase.java:134)
      [junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
      [junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
      [junit] at junit.framework.TestResult.run(TestResult.java:113)
      [junit] at junit.framework.TestCase.run(TestCase.java:124)
      [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243)
      [junit] at junit.framework.TestSuite.run(TestSuite.java:238)
      [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
      [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
      [junit] Exception: Client execution results failed with error code = 1
      [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
      [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.
      [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.)
      [junit] Cleaning up TestCliDriver

      The query that produces the diff is:

      SELECT * FROM SRC DISTRIBUTE BY src.key LIMIT 1

      The query produces nondeterministic results without and ORDER BY clause.

      1. HIVE-2789.D1647.2.patch
        6 kB
        Carl Steinbach
      2. HIVE-2789.D1647.1.patch
        6 kB
        Carl Steinbach
      3. HIVE-2789.3.patch.txt
        6 kB
        Zhenxiao Luo
      4. ASF.LICENSE.NOT.GRANTED--HIVE-2789.D1647.2.patch
        6 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--HIVE-2789.D1647.1.patch
        6 kB
        Phabricator

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2789. query_properties.q contains non-deterministic queries (Zhenxiao Luo via cws) (Revision 1370982)

        Result = ABORTED
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370982
        Files :

        • /hive/trunk/ql/src/test/queries/clientpositive/query_properties.q
        • /hive/trunk/ql/src/test/results/clientpositive/query_properties.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2789 . query_properties.q contains non-deterministic queries (Zhenxiao Luo via cws) (Revision 1370982) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1370982 Files : /hive/trunk/ql/src/test/queries/clientpositive/query_properties.q /hive/trunk/ql/src/test/results/clientpositive/query_properties.q.out
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Zhenxiao!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Zhenxiao!
        Hide
        Carl Steinbach added a comment -

        +1. Will commit if tests pass.

        Show
        Carl Steinbach added a comment - +1. Will commit if tests pass.
        Hide
        Zhenxiao Luo added a comment -

        @Carl: Patch submitted at:

        https://reviews.apache.org/r/5492/

        Thanks,
        Zhenxiao

        Show
        Zhenxiao Luo added a comment - @Carl: Patch submitted at: https://reviews.apache.org/r/5492/ Thanks, Zhenxiao
        Hide
        Carl Steinbach added a comment -

        @Zhenxiao: Please submit a review request on reviews.apache.org. Thanks.

        Show
        Carl Steinbach added a comment - @Zhenxiao: Please submit a review request on reviews.apache.org. Thanks.
        Hide
        Zhenxiao Luo added a comment -

        @Carl and Ashutosh:

        The same bug appears when running query_properties.q in Hive0.9.0 on MapReduce2. I just submitted a new patch for this bug.

        Thanks,
        Zhenxiao

        Show
        Zhenxiao Luo added a comment - @Carl and Ashutosh: The same bug appears when running query_properties.q in Hive0.9.0 on MapReduce2. I just submitted a new patch for this bug. Thanks, Zhenxiao
        Hide
        Ashutosh Chauhan added a comment -

        Patch needs to be rebased.

        Show
        Ashutosh Chauhan added a comment - Patch needs to be rebased.
        Hide
        Ashutosh Chauhan added a comment -

        I still cant apply this patch cleanly either with arc or with patch command directly.

        Show
        Ashutosh Chauhan added a comment - I still cant apply this patch cleanly either with arc or with patch command directly.
        Hide
        Phabricator added a comment -

        cwsteinbach updated the revision "HIVE-2789 [jira] When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result".
        Reviewers: JIRA, kevinwilfong, ashutoshc

        Rebased patch on top of trunk.

        REVISION DETAIL
        https://reviews.facebook.net/D1647

        AFFECTED FILES
        ql/src/test/queries/clientpositive/query_properties.q
        ql/src/test/results/clientpositive/query_properties.q.out

        Show
        Phabricator added a comment - cwsteinbach updated the revision " HIVE-2789 [jira] When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result". Reviewers: JIRA, kevinwilfong, ashutoshc Rebased patch on top of trunk. REVISION DETAIL https://reviews.facebook.net/D1647 AFFECTED FILES ql/src/test/queries/clientpositive/query_properties.q ql/src/test/results/clientpositive/query_properties.q.out
        Hide
        Ashutosh Chauhan added a comment -

        Carl,
        Patch failed to apply cleanly for me. I have reviewed it. Looks good. Can you update the patch, test and commit?

        Show
        Ashutosh Chauhan added a comment - Carl, Patch failed to apply cleanly for me. I have reviewed it. Looks good. Can you update the patch, test and commit?
        Hide
        Phabricator added a comment -

        ashutoshc has accepted the revision "HIVE-2789 [jira] When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result".

        +1 will commit if tests pass.

        REVISION DETAIL
        https://reviews.facebook.net/D1647

        BRANCH
        HIVE-2789-query-properties

        Show
        Phabricator added a comment - ashutoshc has accepted the revision " HIVE-2789 [jira] When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result". +1 will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D1647 BRANCH HIVE-2789 -query-properties
        Hide
        Phabricator added a comment -

        cwsteinbach requested code review of "HIVE-2789 [jira] When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result".
        Reviewers: JIRA

        Fix nondeterminism in query_properties.q by limiting all result sets to 0 rows.

        query_properties.q test failure:

        [junit] Begin query: query_properties.q
        [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 17:00:27 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 17:00:27 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] 12/01/23 17:00:31 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
        [junit] 12/01/23 17:00:31 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
        [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo]wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.][.][.] [0-9]* more -I job_[0-9]_[0-9] -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/query_properties.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/query_properties.q.out
        [junit] 91c91
        [junit] < 97 val_97
        [junit]
        [junit] > 238 val_238
        [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1
        [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.
        [junit] at junit.framework.Assert.fail(Assert.java:50)
        [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_query_properties(TestCliDriver.java:227)
        [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        [junit] at java.lang.reflect.Method.invoke(Method.java:616)
        [junit] at junit.framework.TestCase.runTest(TestCase.java:168)
        [junit] at junit.framework.TestCase.runBare(TestCase.java:134)
        [junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
        [junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
        [junit] at junit.framework.TestResult.run(TestResult.java:113)
        [junit] at junit.framework.TestCase.run(TestCase.java:124)
        [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243)
        [junit] at junit.framework.TestSuite.run(TestSuite.java:238)
        [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
        [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
        [junit] Exception: Client execution results failed with error code = 1
        [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
        [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.
        [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.)
        [junit] Cleaning up TestCliDriver

        The query that produces the diff is:

        SELECT * FROM SRC DISTRIBUTE BY src.key LIMIT 1

        The query produces nondeterministic results without and ORDER BY clause.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1647

        AFFECTED FILES
        ql/src/test/queries/clientpositive/query_properties.q
        ql/src/test/results/clientpositive/query_properties.q.out

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/3507/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - cwsteinbach requested code review of " HIVE-2789 [jira] When integrating into MapReduce2, 'select * from src distribute by src.key limit 1' returns non-deterministic result". Reviewers: JIRA Fix nondeterminism in query_properties.q by limiting all result sets to 0 rows. query_properties.q test failure: [junit] Begin query: query_properties.q [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:13 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:18 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:22 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:27 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:32 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:36 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:41 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:46 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:50 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:55 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 16:59:59 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:04 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:08 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:13 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:18 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:22 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:27 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:27 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] 12/01/23 17:00:31 WARN conf.Configuration: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir [junit] 12/01/23 17:00:31 WARN conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir [junit] diff -a -I file: -I pfile: -I hdfs: -I /tmp/ -I invalidscheme: -I lastUpdateTime -I lastAccessTime -I [Oo] wner -I CreateTime -I LastAccessTime -I Location -I LOCATION ' -I transient_lastDdlTime -I last_modified_ -I java.lang.RuntimeException -I at org -I at sun -I at java -I at junit -I Caused by: -I LOCK_QUERYID: -I LOCK_TIME: -I grantTime -I [.] [.] [.] [0-9] * more -I job_ [0-9] _ [0-9] -I USING 'java -cp /home/cloudera/Code/hive/build/ql/test/logs/clientpositive/query_properties.q.out /home/cloudera/Code/hive/ql/src/test/results/clientpositive/query_properties.q.out [junit] 91c91 [junit] < 97 val_97 [junit] — [junit] > 238 val_238 [junit] junit.framework.AssertionFailedError: Client execution results failed with error code = 1 [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] at junit.framework.Assert.fail(Assert.java:50) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_query_properties(TestCliDriver.java:227) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] at java.lang.reflect.Method.invoke(Method.java:616) [junit] at junit.framework.TestCase.runTest(TestCase.java:168) [junit] at junit.framework.TestCase.runBare(TestCase.java:134) [junit] at junit.framework.TestResult$1.protect(TestResult.java:110) [junit] at junit.framework.TestResult.runProtected(TestResult.java:128) [junit] at junit.framework.TestResult.run(TestResult.java:113) [junit] at junit.framework.TestCase.run(TestCase.java:124) [junit] at junit.framework.TestSuite.runTest(TestSuite.java:243) [junit] at junit.framework.TestSuite.run(TestSuite.java:238) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] Exception: Client execution results failed with error code = 1 [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. [junit] See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs.) [junit] Cleaning up TestCliDriver The query that produces the diff is: SELECT * FROM SRC DISTRIBUTE BY src.key LIMIT 1 The query produces nondeterministic results without and ORDER BY clause. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1647 AFFECTED FILES ql/src/test/queries/clientpositive/query_properties.q ql/src/test/results/clientpositive/query_properties.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3507/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          People

          • Assignee:
            Zhenxiao Luo
            Reporter:
            Zhenxiao Luo
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development