Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.13.0
-
None
Description
HIVE-5356 changed the behavior of some of the arithmetic operations, and the change is not backward compatible, as pointed out in this jira comment
int / int => decimal float / float => double float * float => double float + float => double
Attachments
Attachments
- HIVE-6012.1.patch
- 25 kB
- Jason Dere
- HIVE-6012.2.patch
- 1.06 MB
- Jason Dere
- HIVE-6012.3.patch
- 1.06 MB
- Jason Dere
- HIVE-6012.4.patch
- 1.07 MB
- Jason Dere
- HIVE-6012.5.patch
- 783 kB
- Jason Dere
- HIVE-6012.6.patch
- 783 kB
- Jason Dere
Issue Links
- is related to
-
HIVE-5356 Move arithmatic UDFs to generic UDF implementations
- Resolved
Activity
Taking a look at this. Listed below are the numeric operations that changed when they were converted to use GenericUDFs. For these operations the return type will be reverted back to the Hive-0.12 behavior:
int / int -> decimal, used to be double in Hive-0.12
float + int -> double, used to be float in Hive-0.12
float + float -> double, used to be float in Hive-0.12
float * int -> double, used to be float in Hive-0.12
float * float -> double, used to be float in Hive-0.12
decimal + float -> double, used to be decimal in Hive-0.12
decimal + double -> double, used to be decimal in Hive-0.12
decimal * float -> double, used to be decimal in Hive-0.12
decimal * double -> double, used to be decimal in Hive-0.12
decimal / float -> double, used to be decimal in Hive-0.12
decimal / double -> double, used to be decimal in Hive-0.12
Reverting the last 6 operations, involving decimal and float/double inputs, to return decimal type is a bit problematic, due to the fact that we would need to determine a precision/scale for the result type. The float/double inputs would need to be converted to a decimal value and it is not guaranteed that the value can be represented by a decimal, if the float/decimal value is too large. Have a couple of options for what to do, if there is any feedback let me know.
1) Revert the type to return decimal, and come up with some arbitrary precision/scale for the result type, such as the system precison/scale of 38/18. This means a bit more special case code, and a chance that we may not be able to represent the float/double portion if the value is larger than the precision.
2) Given the large changes in the decimal type in this current release, make the call that it is ok to leave the changes in result type for the numeric operations involving decimal inputs, and consider these as part of the changes related to decimal types. This is the more SQL compliant behavior. Currently leaning toward this approach.
Also will add config setting hive.sql.ansi to be able to switch this return type behavior from the backward compatible mode to the more SQL compliant return types that were done in HIVE-5356.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12624984/HIVE-6012.1.patch
ERROR: -1 due to 93 failed/errored test(s), 4952 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.initializationError org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_insert_common_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag_queries org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_num_op_type_conv org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_empty_strings org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reduce_deduplicate_extended org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_timestamp_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_pmod org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_uniquejoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_not org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_windowspec org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hadoop.hive.ql.parse.TestParse.testParse_input8
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1015/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 93 tests failed
This message is automatically generated.
ATTACHMENT ID: 12624984
patch v2:
- Since SessionState not available during map/reduce tasks, config setting should be read and saved during query compilation by Hive client, plan serialization will make the values available in map/reduce tasks.
- update diffs
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12625616/HIVE-6012.2.patch
ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed
Failed tests:
org.apache.hadoop.hive.ql.exec.TestExpressionEvaluator.testExprNodeSpeed
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1072/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed
This message is automatically generated.
ATTACHMENT ID: 12625616
Please document hive.sql.ansi in hive-default.xml.template, then I'll put it in the Configuration Properties wikidoc.
When this is committed, details should also be documented in the wiki. The int & float operations go in the Arithmetic Operators table in Operators and UDFs. Decimal operations could go in the Decimal Types section of the Types doc, unless you want to keep all this information together and just have a link from the Types doc to the Arithmetic Operators table.
Quick reference:
Greetings night owl, maybe you need to rest – you've attached the wrong file. Patch 3 contains a bunch of git commit records. (Hoo hoo hoo ... wha?)
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12625909/HIVE-6012.3.patch
ERROR: -1 due to 1 failed/errored test(s), 4980 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1100/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed
This message is automatically generated.
ATTACHMENT ID: 12625909
There are some practical problems with introducing a hive.sql.ansi flag. Lets say a hive user turns it to true in hive 0.13 and write several queries using hive 0.13. Then in hive 0.14 we find some other desirable standards compatible behavior (but backward incompatible with earlier versions of hive) and we control that using same flag. When the user upgrades to hive 0.14, the user queries can break with either values for the flag.
I think it is better to have a more specific flag, say hive.sql.ansi.arithmetic
Since the behavior of float arithmetic in hive 0.12 was SQL compliant, I think we can go back to old behavior without this config option. There more configurable the behavior is, there more untested configurations there will be.
A more specific config flag makes sense. I'll see about trying to rework the patch.
Had some offline discussion about the config setting to use here. Suggestion was to use a single config setting (say, hive.compat) to control backward compatibility behavior. Rather than make the config setting a simple true/false switch, the config setting would be a string to indicate what version of Hive behavior to support (0.12, 0.13, etc). So for Thejas' example in his comment, setting hive.compat=0.12 would set int / int => double. If there were additional (configurable) behavior changes added in 0.14, they could be disabled by setting hive.compat=0.13, or even back to 0.12 if the user still wants the old int / int behavior.
I'll try to take a crack at this.
Overall: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12626821/HIVE-6012.4.patch
SUCCESS: +1 5006 tests passed
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1176/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase
This message is automatically generated.
ATTACHMENT ID: 12626821
Thanks for looking; I need to rebase this patch with trunk, in process right now.
Overall: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633756/HIVE-6012.6.patch
ERROR: -1 due to 1 failed/errored test(s), 5391 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1708/console
Messages:
Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed
This message is automatically generated.
ATTACHMENT ID: 12633756
Doc questions: Does hive.compat accept any Hive release number, or only some? The default is 0.12 but can it be set to 0.12.0? Would it balk at 0.12.5 or just ignore .5? What about 0.11 and earlier releases – would they give the same behavior as 0.12?
Based on exact string match with the supported release numbers (0.12, 0.13, or "latest" to use latest supported level). If it does not recognize number it will squawk error and use default support level.
Test case Minimr bucketizedhiveinputformat.q passes for me when run locally, on both Mac/Linux. Flaky test?
Review request: hive.compat is documented in the wiki and needs to be reviewed carefully. For example, does the parameter affect other operations besides division of ints?
Still to come (pending answer to question above): documenting hive.compat in the table of arithmetic operators:
Just to be crystal clear: so hive.compat only affects division of ints?
Thanks jdere.
I started a dev alias thread about having ANSI flag to choose between old Hive mode and ANSI SQL mode