Hive
  1. Hive
  2. HIVE-6012

restore backward compatibility of arithmetic operations

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      HIVE-5356 changed the behavior of some of the arithmetic operations, and the change is not backward compatible, as pointed out in this jira comment

      int / int => decimal
      float / float => double
      float * float => double
      float + float => double
      
      1. HIVE-6012.6.patch
        783 kB
        Jason Dere
      2. HIVE-6012.5.patch
        783 kB
        Jason Dere
      3. HIVE-6012.4.patch
        1.07 MB
        Jason Dere
      4. HIVE-6012.3.patch
        1.06 MB
        Jason Dere
      5. HIVE-6012.2.patch
        1.06 MB
        Jason Dere
      6. HIVE-6012.1.patch
        25 kB
        Jason Dere

        Issue Links

          Activity

          Hide
          Jason Dere added a comment -

          Committed to 0.13 and trunk.

          Show
          Jason Dere added a comment - Committed to 0.13 and trunk.
          Hide
          Jason Dere added a comment -

          Test case Minimr bucketizedhiveinputformat.q passes for me when run locally, on both Mac/Linux. Flaky test?

          Show
          Jason Dere added a comment - Test case Minimr bucketizedhiveinputformat.q passes for me when run locally, on both Mac/Linux. Flaky test?
          Hide
          Jason Dere added a comment -

          Based on exact string match with the supported release numbers (0.12, 0.13, or "latest" to use latest supported level). If it does not recognize number it will squawk error and use default support level.

          Show
          Jason Dere added a comment - Based on exact string match with the supported release numbers (0.12, 0.13, or "latest" to use latest supported level). If it does not recognize number it will squawk error and use default support level.
          Hide
          Lefty Leverenz added a comment -

          Doc questions: Does hive.compat accept any Hive release number, or only some? The default is 0.12 but can it be set to 0.12.0? Would it balk at 0.12.5 or just ignore .5? What about 0.11 and earlier releases – would they give the same behavior as 0.12?

          Show
          Lefty Leverenz added a comment - Doc questions: Does hive.compat accept any Hive release number, or only some? The default is 0.12 but can it be set to 0.12.0? Would it balk at 0.12.5 or just ignore .5? What about 0.11 and earlier releases – would they give the same behavior as 0.12?
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12633756/HIVE-6012.6.patch

          ERROR: -1 due to 1 failed/errored test(s), 5391 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12633756

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12633756/HIVE-6012.6.patch ERROR: -1 due to 1 failed/errored test(s), 5391 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12633756
          Hide
          Jason Dere added a comment -

          still no precommit test results .. uploading patch again

          Show
          Jason Dere added a comment - still no precommit test results .. uploading patch again
          Hide
          Gunther Hagleitner added a comment -

          +1

          Show
          Gunther Hagleitner added a comment - +1
          Hide
          Jason Dere added a comment -

          rebase with trunk

          Show
          Jason Dere added a comment - rebase with trunk
          Hide
          Jason Dere added a comment -

          Thanks for looking; I need to rebase this patch with trunk, in process right now.

          Show
          Jason Dere added a comment - Thanks for looking; I need to rebase this patch with trunk, in process right now.
          Hide
          Sergey Shelukhin added a comment -

          lgtm; some small comment on RB

          Show
          Sergey Shelukhin added a comment - lgtm; some small comment on RB
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626821/HIVE-6012.4.patch

          SUCCESS: +1 5006 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12626821

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626821/HIVE-6012.4.patch SUCCESS: +1 5006 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12626821
          Hide
          Jason Dere added a comment -

          patch v4: rebased, use hive.compat setting for int division behavior

          Show
          Jason Dere added a comment - patch v4: rebased, use hive.compat setting for int division behavior
          Hide
          Jason Dere added a comment -

          Had some offline discussion about the config setting to use here. Suggestion was to use a single config setting (say, hive.compat) to control backward compatibility behavior. Rather than make the config setting a simple true/false switch, the config setting would be a string to indicate what version of Hive behavior to support (0.12, 0.13, etc). So for Thejas' example in his comment, setting hive.compat=0.12 would set int / int => double. If there were additional (configurable) behavior changes added in 0.14, they could be disabled by setting hive.compat=0.13, or even back to 0.12 if the user still wants the old int / int behavior.
          I'll try to take a crack at this.

          Show
          Jason Dere added a comment - Had some offline discussion about the config setting to use here. Suggestion was to use a single config setting (say, hive.compat) to control backward compatibility behavior. Rather than make the config setting a simple true/false switch, the config setting would be a string to indicate what version of Hive behavior to support (0.12, 0.13, etc). So for Thejas' example in his comment, setting hive.compat=0.12 would set int / int => double. If there were additional (configurable) behavior changes added in 0.14, they could be disabled by setting hive.compat=0.13, or even back to 0.12 if the user still wants the old int / int behavior. I'll try to take a crack at this.
          Hide
          Jason Dere added a comment -

          A more specific config flag makes sense. I'll see about trying to rework the patch.

          Show
          Jason Dere added a comment - A more specific config flag makes sense. I'll see about trying to rework the patch.
          Hide
          Thejas M Nair added a comment -

          There are some practical problems with introducing a hive.sql.ansi flag. Lets say a hive user turns it to true in hive 0.13 and write several queries using hive 0.13. Then in hive 0.14 we find some other desirable standards compatible behavior (but backward incompatible with earlier versions of hive) and we control that using same flag. When the user upgrades to hive 0.14, the user queries can break with either values for the flag.

          I think it is better to have a more specific flag, say hive.sql.ansi.arithmetic

          Since the behavior of float arithmetic in hive 0.12 was SQL compliant, I think we can go back to old behavior without this config option. There more configurable the behavior is, there more untested configurations there will be.

          Show
          Thejas M Nair added a comment - There are some practical problems with introducing a hive.sql.ansi flag. Lets say a hive user turns it to true in hive 0.13 and write several queries using hive 0.13. Then in hive 0.14 we find some other desirable standards compatible behavior (but backward incompatible with earlier versions of hive) and we control that using same flag. When the user upgrades to hive 0.14, the user queries can break with either values for the flag. I think it is better to have a more specific flag, say hive.sql.ansi.arithmetic Since the behavior of float arithmetic in hive 0.12 was SQL compliant, I think we can go back to old behavior without this config option. There more configurable the behavior is, there more untested configurations there will be.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12625909/HIVE-6012.3.patch

          ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12625909

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12625909/HIVE-6012.3.patch ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12625909
          Hide
          Jason Dere added a comment -

          Oops! Didn't create diff properly, uploading patch v3 again.

          Show
          Jason Dere added a comment - Oops! Didn't create diff properly, uploading patch v3 again.
          Hide
          Lefty Leverenz added a comment -

          Greetings night owl, maybe you need to rest – you've attached the wrong file. Patch 3 contains a bunch of git commit records. (Hoo hoo hoo ... wha?)

          Show
          Lefty Leverenz added a comment - Greetings night owl, maybe you need to rest – you've attached the wrong file. Patch 3 contains a bunch of git commit records. (Hoo hoo hoo ... wha?)
          Hide
          Jason Dere added a comment -

          fix remaining test failure, update hive-default.xml.template

          Show
          Jason Dere added a comment - fix remaining test failure, update hive-default.xml.template
          Hide
          Lefty Leverenz added a comment -

          Please document hive.sql.ansi in hive-default.xml.template, then I'll put it in the Configuration Properties wikidoc.

          When this is committed, details should also be documented in the wiki. The int & float operations go in the Arithmetic Operators table in Operators and UDFs. Decimal operations could go in the Decimal Types section of the Types doc, unless you want to keep all this information together and just have a link from the Types doc to the Arithmetic Operators table.

          Quick reference:

          Show
          Lefty Leverenz added a comment - Please document hive.sql.ansi in hive-default.xml.template, then I'll put it in the Configuration Properties wikidoc. When this is committed, details should also be documented in the wiki. The int & float operations go in the Arithmetic Operators table in Operators and UDFs. Decimal operations could go in the Decimal Types section of the Types doc, unless you want to keep all this information together and just have a link from the Types doc to the Arithmetic Operators table. Quick reference: Operators and UDFs: Arithmetic Operators Decimal Types: Mathematical UDFs Configuration Properties
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12625616/HIVE-6012.2.patch

          ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed
          Failed tests:

          org.apache.hadoop.hive.ql.exec.TestExpressionEvaluator.testExprNodeSpeed
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12625616

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12625616/HIVE-6012.2.patch ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed Failed tests: org.apache.hadoop.hive.ql.exec.TestExpressionEvaluator.testExprNodeSpeed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12625616
          Hide
          Jason Dere added a comment -

          patch v2:

          • Since SessionState not available during map/reduce tasks, config setting should be read and saved during query compilation by Hive client, plan serialization will make the values available in map/reduce tasks.
          • update diffs
          Show
          Jason Dere added a comment - patch v2: Since SessionState not available during map/reduce tasks, config setting should be read and saved during query compilation by Hive client, plan serialization will make the values available in map/reduce tasks. update diffs
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12624984/HIVE-6012.1.patch

          ERROR: -1 due to 93 failed/errored test(s), 4952 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.initializationError
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_insert_common_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag_queries
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_num_op_type_conv
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_empty_strings
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_extended
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_timestamp_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_pmod
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_uniquejoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_0
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_not
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_windowspec
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
          org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_num_buckets
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative
          org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
          org.apache.hadoop.hive.ql.parse.TestParse.testParse_input8
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 93 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12624984

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12624984/HIVE-6012.1.patch ERROR: -1 due to 93 failed/errored test(s), 4952 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.initializationError org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_insert_common_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag_queries org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_num_op_type_conv org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_empty_strings org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_extended org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_timestamp_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_pmod org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_uniquejoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_not org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_windowspec org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hadoop.hive.ql.parse.TestParse.testParse_input8 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 93 tests failed This message is automatically generated. ATTACHMENT ID: 12624984
          Show
          Jason Dere added a comment - https://reviews.apache.org/r/17302/
          Hide
          Jason Dere added a comment -

          Patch v1

          Show
          Jason Dere added a comment - Patch v1
          Hide
          Sergey Shelukhin added a comment -

          +1 for sql.ansi flag

          Show
          Sergey Shelukhin added a comment - +1 for sql.ansi flag
          Hide
          Jason Dere added a comment -

          Also will add config setting hive.sql.ansi to be able to switch this return type behavior from the backward compatible mode to the more SQL compliant return types that were done in HIVE-5356.

          Show
          Jason Dere added a comment - Also will add config setting hive.sql.ansi to be able to switch this return type behavior from the backward compatible mode to the more SQL compliant return types that were done in HIVE-5356 .
          Hide
          Jason Dere added a comment -

          Taking a look at this. Listed below are the numeric operations that changed when they were converted to use GenericUDFs. For these operations the return type will be reverted back to the Hive-0.12 behavior:

          int / int -> decimal, used to be double in Hive-0.12
          float + int -> double, used to be float in Hive-0.12
          float + float -> double, used to be float in Hive-0.12
          float * int -> double, used to be float in Hive-0.12
          float * float -> double, used to be float in Hive-0.12

          decimal + float -> double, used to be decimal in Hive-0.12
          decimal + double -> double, used to be decimal in Hive-0.12
          decimal * float -> double, used to be decimal in Hive-0.12
          decimal * double -> double, used to be decimal in Hive-0.12
          decimal / float -> double, used to be decimal in Hive-0.12
          decimal / double -> double, used to be decimal in Hive-0.12

          Reverting the last 6 operations, involving decimal and float/double inputs, to return decimal type is a bit problematic, due to the fact that we would need to determine a precision/scale for the result type. The float/double inputs would need to be converted to a decimal value and it is not guaranteed that the value can be represented by a decimal, if the float/decimal value is too large. Have a couple of options for what to do, if there is any feedback let me know.
          1) Revert the type to return decimal, and come up with some arbitrary precision/scale for the result type, such as the system precison/scale of 38/18. This means a bit more special case code, and a chance that we may not be able to represent the float/double portion if the value is larger than the precision.
          2) Given the large changes in the decimal type in this current release, make the call that it is ok to leave the changes in result type for the numeric operations involving decimal inputs, and consider these as part of the changes related to decimal types. This is the more SQL compliant behavior. Currently leaning toward this approach.

          Show
          Jason Dere added a comment - Taking a look at this. Listed below are the numeric operations that changed when they were converted to use GenericUDFs. For these operations the return type will be reverted back to the Hive-0.12 behavior: int / int -> decimal, used to be double in Hive-0.12 float + int -> double, used to be float in Hive-0.12 float + float -> double, used to be float in Hive-0.12 float * int -> double, used to be float in Hive-0.12 float * float -> double, used to be float in Hive-0.12 decimal + float -> double, used to be decimal in Hive-0.12 decimal + double -> double, used to be decimal in Hive-0.12 decimal * float -> double, used to be decimal in Hive-0.12 decimal * double -> double, used to be decimal in Hive-0.12 decimal / float -> double, used to be decimal in Hive-0.12 decimal / double -> double, used to be decimal in Hive-0.12 Reverting the last 6 operations, involving decimal and float/double inputs, to return decimal type is a bit problematic, due to the fact that we would need to determine a precision/scale for the result type. The float/double inputs would need to be converted to a decimal value and it is not guaranteed that the value can be represented by a decimal, if the float/decimal value is too large. Have a couple of options for what to do, if there is any feedback let me know. 1) Revert the type to return decimal, and come up with some arbitrary precision/scale for the result type, such as the system precison/scale of 38/18. This means a bit more special case code, and a chance that we may not be able to represent the float/double portion if the value is larger than the precision. 2) Given the large changes in the decimal type in this current release, make the call that it is ok to leave the changes in result type for the numeric operations involving decimal inputs, and consider these as part of the changes related to decimal types. This is the more SQL compliant behavior. Currently leaning toward this approach.
          Hide
          Sergey Shelukhin added a comment -

          I started a dev alias thread about having ANSI flag to choose between old Hive mode and ANSI SQL mode

          Show
          Sergey Shelukhin added a comment - I started a dev alias thread about having ANSI flag to choose between old Hive mode and ANSI SQL mode

            People

            • Assignee:
              Jason Dere
              Reporter:
              Thejas M Nair
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development