Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6012

restore backward compatibility of arithmetic operations

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.13.0
    • 0.13.0
    • Query Processor
    • None

    Description

      HIVE-5356 changed the behavior of some of the arithmetic operations, and the change is not backward compatible, as pointed out in this jira comment

      int / int => decimal
      float / float => double
      float * float => double
      float + float => double
      

      Attachments

        1. HIVE-6012.1.patch
          25 kB
          Jason Dere
        2. HIVE-6012.2.patch
          1.06 MB
          Jason Dere
        3. HIVE-6012.3.patch
          1.06 MB
          Jason Dere
        4. HIVE-6012.4.patch
          1.07 MB
          Jason Dere
        5. HIVE-6012.5.patch
          783 kB
          Jason Dere
        6. HIVE-6012.6.patch
          783 kB
          Jason Dere

        Issue Links

          Activity

            I started a dev alias thread about having ANSI flag to choose between old Hive mode and ANSI SQL mode

            sershe Sergey Shelukhin added a comment - I started a dev alias thread about having ANSI flag to choose between old Hive mode and ANSI SQL mode
            jdere Jason Dere added a comment -

            Taking a look at this. Listed below are the numeric operations that changed when they were converted to use GenericUDFs. For these operations the return type will be reverted back to the Hive-0.12 behavior:

            int / int -> decimal, used to be double in Hive-0.12
            float + int -> double, used to be float in Hive-0.12
            float + float -> double, used to be float in Hive-0.12
            float * int -> double, used to be float in Hive-0.12
            float * float -> double, used to be float in Hive-0.12

            decimal + float -> double, used to be decimal in Hive-0.12
            decimal + double -> double, used to be decimal in Hive-0.12
            decimal * float -> double, used to be decimal in Hive-0.12
            decimal * double -> double, used to be decimal in Hive-0.12
            decimal / float -> double, used to be decimal in Hive-0.12
            decimal / double -> double, used to be decimal in Hive-0.12

            Reverting the last 6 operations, involving decimal and float/double inputs, to return decimal type is a bit problematic, due to the fact that we would need to determine a precision/scale for the result type. The float/double inputs would need to be converted to a decimal value and it is not guaranteed that the value can be represented by a decimal, if the float/decimal value is too large. Have a couple of options for what to do, if there is any feedback let me know.
            1) Revert the type to return decimal, and come up with some arbitrary precision/scale for the result type, such as the system precison/scale of 38/18. This means a bit more special case code, and a chance that we may not be able to represent the float/double portion if the value is larger than the precision.
            2) Given the large changes in the decimal type in this current release, make the call that it is ok to leave the changes in result type for the numeric operations involving decimal inputs, and consider these as part of the changes related to decimal types. This is the more SQL compliant behavior. Currently leaning toward this approach.

            jdere Jason Dere added a comment - Taking a look at this. Listed below are the numeric operations that changed when they were converted to use GenericUDFs. For these operations the return type will be reverted back to the Hive-0.12 behavior: int / int -> decimal, used to be double in Hive-0.12 float + int -> double, used to be float in Hive-0.12 float + float -> double, used to be float in Hive-0.12 float * int -> double, used to be float in Hive-0.12 float * float -> double, used to be float in Hive-0.12 decimal + float -> double, used to be decimal in Hive-0.12 decimal + double -> double, used to be decimal in Hive-0.12 decimal * float -> double, used to be decimal in Hive-0.12 decimal * double -> double, used to be decimal in Hive-0.12 decimal / float -> double, used to be decimal in Hive-0.12 decimal / double -> double, used to be decimal in Hive-0.12 Reverting the last 6 operations, involving decimal and float/double inputs, to return decimal type is a bit problematic, due to the fact that we would need to determine a precision/scale for the result type. The float/double inputs would need to be converted to a decimal value and it is not guaranteed that the value can be represented by a decimal, if the float/decimal value is too large. Have a couple of options for what to do, if there is any feedback let me know. 1) Revert the type to return decimal, and come up with some arbitrary precision/scale for the result type, such as the system precison/scale of 38/18. This means a bit more special case code, and a chance that we may not be able to represent the float/double portion if the value is larger than the precision. 2) Given the large changes in the decimal type in this current release, make the call that it is ok to leave the changes in result type for the numeric operations involving decimal inputs, and consider these as part of the changes related to decimal types. This is the more SQL compliant behavior. Currently leaning toward this approach.
            jdere Jason Dere added a comment -

            Also will add config setting hive.sql.ansi to be able to switch this return type behavior from the backward compatible mode to the more SQL compliant return types that were done in HIVE-5356.

            jdere Jason Dere added a comment - Also will add config setting hive.sql.ansi to be able to switch this return type behavior from the backward compatible mode to the more SQL compliant return types that were done in HIVE-5356 .

            +1 for sql.ansi flag

            sershe Sergey Shelukhin added a comment - +1 for sql.ansi flag
            jdere Jason Dere added a comment -

            Patch v1

            jdere Jason Dere added a comment - Patch v1
            jdere Jason Dere added a comment - https://reviews.apache.org/r/17302/
            hiveqa Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12624984/HIVE-6012.1.patch

            ERROR: -1 due to 93 failed/errored test(s), 4952 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestCliDriver.initializationError
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets1
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_insert_common_distinct
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer3
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag_queries
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce1
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce2
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce3
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce4
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce5
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce6
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce7
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce8
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition4
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition5
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_num_op_type_conv
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_empty_strings
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat10
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat11
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat15
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat16
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat8
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat9
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_extended
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook_hadoop20
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_timestamp_3
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_pmod
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_uniquejoin
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_0
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_1
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_12
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_2
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_3
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_4
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_5
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_not
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_windowspec
            org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
            org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table
            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_num_buckets
            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2
            org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative
            org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
            org.apache.hadoop.hive.ql.parse.TestParse.testParse_input8
            

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 93 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12624984

            hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12624984/HIVE-6012.1.patch ERROR: -1 due to 93 failed/errored test(s), 4952 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.initializationError org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_insert_common_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag_queries org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_num_op_type_conv org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_empty_strings org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_extended org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_timestamp_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_pmod org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_uniquejoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_not org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_windowspec org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hadoop.hive.ql.parse.TestParse.testParse_input8 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 93 tests failed This message is automatically generated. ATTACHMENT ID: 12624984
            jdere Jason Dere added a comment -

            patch v2:

            • Since SessionState not available during map/reduce tasks, config setting should be read and saved during query compilation by Hive client, plan serialization will make the values available in map/reduce tasks.
            • update diffs
            jdere Jason Dere added a comment - patch v2: Since SessionState not available during map/reduce tasks, config setting should be read and saved during query compilation by Hive client, plan serialization will make the values available in map/reduce tasks. update diffs
            hiveqa Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12625616/HIVE-6012.2.patch

            ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed
            Failed tests:

            org.apache.hadoop.hive.ql.exec.TestExpressionEvaluator.testExprNodeSpeed
            

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 1 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12625616

            hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12625616/HIVE-6012.2.patch ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed Failed tests: org.apache.hadoop.hive.ql.exec.TestExpressionEvaluator.testExprNodeSpeed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12625616
            leftyl Lefty Leverenz added a comment -

            Please document hive.sql.ansi in hive-default.xml.template, then I'll put it in the Configuration Properties wikidoc.

            When this is committed, details should also be documented in the wiki. The int & float operations go in the Arithmetic Operators table in Operators and UDFs. Decimal operations could go in the Decimal Types section of the Types doc, unless you want to keep all this information together and just have a link from the Types doc to the Arithmetic Operators table.

            Quick reference:

            leftyl Lefty Leverenz added a comment - Please document hive.sql.ansi in hive-default.xml.template, then I'll put it in the Configuration Properties wikidoc. When this is committed, details should also be documented in the wiki. The int & float operations go in the Arithmetic Operators table in Operators and UDFs. Decimal operations could go in the Decimal Types section of the Types doc, unless you want to keep all this information together and just have a link from the Types doc to the Arithmetic Operators table. Quick reference: Operators and UDFs: Arithmetic Operators Decimal Types: Mathematical UDFs Configuration Properties
            jdere Jason Dere added a comment -

            fix remaining test failure, update hive-default.xml.template

            jdere Jason Dere added a comment - fix remaining test failure, update hive-default.xml.template
            leftyl Lefty Leverenz added a comment -

            Greetings night owl, maybe you need to rest – you've attached the wrong file. Patch 3 contains a bunch of git commit records. (Hoo hoo hoo ... wha?)

            leftyl Lefty Leverenz added a comment - Greetings night owl, maybe you need to rest – you've attached the wrong file. Patch 3 contains a bunch of git commit records. (Hoo hoo hoo ... wha?)
            jdere Jason Dere added a comment -

            Oops! Didn't create diff properly, uploading patch v3 again.

            jdere Jason Dere added a comment - Oops! Didn't create diff properly, uploading patch v3 again.
            hiveqa Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12625909/HIVE-6012.3.patch

            ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
            

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 1 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12625909

            hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12625909/HIVE-6012.3.patch ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12625909
            thejas Thejas Nair added a comment -

            There are some practical problems with introducing a hive.sql.ansi flag. Lets say a hive user turns it to true in hive 0.13 and write several queries using hive 0.13. Then in hive 0.14 we find some other desirable standards compatible behavior (but backward incompatible with earlier versions of hive) and we control that using same flag. When the user upgrades to hive 0.14, the user queries can break with either values for the flag.

            I think it is better to have a more specific flag, say hive.sql.ansi.arithmetic

            Since the behavior of float arithmetic in hive 0.12 was SQL compliant, I think we can go back to old behavior without this config option. There more configurable the behavior is, there more untested configurations there will be.

            thejas Thejas Nair added a comment - There are some practical problems with introducing a hive.sql.ansi flag. Lets say a hive user turns it to true in hive 0.13 and write several queries using hive 0.13. Then in hive 0.14 we find some other desirable standards compatible behavior (but backward incompatible with earlier versions of hive) and we control that using same flag. When the user upgrades to hive 0.14, the user queries can break with either values for the flag. I think it is better to have a more specific flag, say hive.sql.ansi.arithmetic Since the behavior of float arithmetic in hive 0.12 was SQL compliant, I think we can go back to old behavior without this config option. There more configurable the behavior is, there more untested configurations there will be.
            jdere Jason Dere added a comment -

            A more specific config flag makes sense. I'll see about trying to rework the patch.

            jdere Jason Dere added a comment - A more specific config flag makes sense. I'll see about trying to rework the patch.
            jdere Jason Dere added a comment -

            Had some offline discussion about the config setting to use here. Suggestion was to use a single config setting (say, hive.compat) to control backward compatibility behavior. Rather than make the config setting a simple true/false switch, the config setting would be a string to indicate what version of Hive behavior to support (0.12, 0.13, etc). So for Thejas' example in his comment, setting hive.compat=0.12 would set int / int => double. If there were additional (configurable) behavior changes added in 0.14, they could be disabled by setting hive.compat=0.13, or even back to 0.12 if the user still wants the old int / int behavior.
            I'll try to take a crack at this.

            jdere Jason Dere added a comment - Had some offline discussion about the config setting to use here. Suggestion was to use a single config setting (say, hive.compat) to control backward compatibility behavior. Rather than make the config setting a simple true/false switch, the config setting would be a string to indicate what version of Hive behavior to support (0.12, 0.13, etc). So for Thejas' example in his comment, setting hive.compat=0.12 would set int / int => double. If there were additional (configurable) behavior changes added in 0.14, they could be disabled by setting hive.compat=0.13, or even back to 0.12 if the user still wants the old int / int behavior. I'll try to take a crack at this.
            jdere Jason Dere added a comment -

            patch v4: rebased, use hive.compat setting for int division behavior

            jdere Jason Dere added a comment - patch v4: rebased, use hive.compat setting for int division behavior
            hiveqa Hive QA added a comment -

            Overall: +1 all checks pass

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12626821/HIVE-6012.4.patch

            SUCCESS: +1 5006 tests passed

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            

            This message is automatically generated.

            ATTACHMENT ID: 12626821

            hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626821/HIVE-6012.4.patch SUCCESS: +1 5006 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12626821

            lgtm; some small comment on RB

            sershe Sergey Shelukhin added a comment - lgtm; some small comment on RB
            jdere Jason Dere added a comment -

            Thanks for looking; I need to rebase this patch with trunk, in process right now.

            jdere Jason Dere added a comment - Thanks for looking; I need to rebase this patch with trunk, in process right now.
            jdere Jason Dere added a comment -

            rebase with trunk

            jdere Jason Dere added a comment - rebase with trunk
            jdere Jason Dere added a comment -

            still no precommit test results .. uploading patch again

            jdere Jason Dere added a comment - still no precommit test results .. uploading patch again
            hiveqa Hive QA added a comment -

            Overall: -1 at least one tests failed

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12633756/HIVE-6012.6.patch

            ERROR: -1 due to 1 failed/errored test(s), 5391 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
            

            Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/testReport
            Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/console

            Messages:

            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 1 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12633756

            hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12633756/HIVE-6012.6.patch ERROR: -1 due to 1 failed/errored test(s), 5391 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12633756
            leftyl Lefty Leverenz added a comment -

            Doc questions: Does hive.compat accept any Hive release number, or only some? The default is 0.12 but can it be set to 0.12.0? Would it balk at 0.12.5 or just ignore .5? What about 0.11 and earlier releases – would they give the same behavior as 0.12?

            leftyl Lefty Leverenz added a comment - Doc questions: Does hive.compat accept any Hive release number, or only some? The default is 0.12 but can it be set to 0.12.0? Would it balk at 0.12.5 or just ignore .5? What about 0.11 and earlier releases – would they give the same behavior as 0.12?
            jdere Jason Dere added a comment -

            Based on exact string match with the supported release numbers (0.12, 0.13, or "latest" to use latest supported level). If it does not recognize number it will squawk error and use default support level.

            jdere Jason Dere added a comment - Based on exact string match with the supported release numbers (0.12, 0.13, or "latest" to use latest supported level). If it does not recognize number it will squawk error and use default support level.
            jdere Jason Dere added a comment -

            Test case Minimr bucketizedhiveinputformat.q passes for me when run locally, on both Mac/Linux. Flaky test?

            jdere Jason Dere added a comment - Test case Minimr bucketizedhiveinputformat.q passes for me when run locally, on both Mac/Linux. Flaky test?
            jdere Jason Dere added a comment -

            Committed to 0.13 and trunk.

            jdere Jason Dere added a comment - Committed to 0.13 and trunk.
            leftyl Lefty Leverenz added a comment -

            Review request: hive.compat is documented in the wiki and needs to be reviewed carefully. For example, does the parameter affect other operations besides division of ints?

            Still to come (pending answer to question above): documenting hive.compat in the table of arithmetic operators:

            leftyl Lefty Leverenz added a comment - Review request: hive.compat is documented in the wiki and needs to be reviewed carefully. For example, does the parameter affect other operations besides division of ints? Configuration Properties – hive.compat Still to come (pending answer to question above): documenting hive.compat in the table of arithmetic operators: Hive Operators and UDFs – Arithmetic Operators
            leftyl Lefty Leverenz added a comment -

            Pinging jdere for doc review and more information.

            leftyl Lefty Leverenz added a comment - Pinging jdere for doc review and more information.
            jdere Jason Dere added a comment -

            Yes leftylev that looks correct.

            jdere Jason Dere added a comment - Yes leftylev that looks correct.
            leftyl Lefty Leverenz added a comment -

            Just to be crystal clear: so hive.compat only affects division of ints?

            Thanks jdere.

            leftyl Lefty Leverenz added a comment - Just to be crystal clear: so hive.compat only affects division of ints? Thanks jdere .
            jdere Jason Dere added a comment -

            Yes that's currently the only setting affected by hive.compat.

            jdere Jason Dere added a comment - Yes that's currently the only setting affected by hive.compat.
            leftyl Lefty Leverenz added a comment - Doc done: Configuration Properties – hive.compat Hive Operators and UDFs – Arithmetic Operators – A / B

            People

              jdere Jason Dere
              thejas Thejas Nair
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: