Hive
  1. Hive
  2. HIVE-2427

Warn user that precision is lost when bigint is implicitly cast to double in joins and unions.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When a bigint is implicitly cast to a double (when a bigint is involved in an equality expression with a string or double as part of a join condition, or when a bigint column is to be combined with a string or double column as part of a union) precision may be lost, resulting in unexpected behavior. Until we fix the underlying issue we should throw an error in strict mode, and a warning in nonstrict mode alerting the user about this.

      1. HIVE-2427.5.patch.txt
        19 kB
        Kevin Wilfong
      2. HIVE-2427.3.patch.txt
        23 kB
        Kevin Wilfong
      3. HIVE-2427.2.patch.txt
        26 kB
        Kevin Wilfong
      4. HIVE-2427.1.patch.txt
        25 kB
        Kevin Wilfong
      5. ASF.LICENSE.NOT.GRANTED--HIVE-2427.D2181.1.patch
        23 kB
        Phabricator

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1705/
        -----------------------------------------------------------

        Review request for hive and Siying Dong.

        Summary
        -------

        I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double.

        I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode.

        This addresses bug HIVE-2427.
        https://issues.apache.org/jira/browse/HIVE-2427

        Diffs


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293
        trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293
        trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293
        trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293
        trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293

        Diff: https://reviews.apache.org/r/1705/diff

        Testing
        -------

        I added several more tests to demonstrate the errors.

        I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1705/ ----------------------------------------------------------- Review request for hive and Siying Dong. Summary ------- I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double. I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode. This addresses bug HIVE-2427 . https://issues.apache.org/jira/browse/HIVE-2427 Diffs trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293 Diff: https://reviews.apache.org/r/1705/diff Testing ------- I added several more tests to demonstrate the errors. I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1705/
        -----------------------------------------------------------

        (Updated 2011-09-06 17:41:24.308161)

        Review request for hive and Siying Dong.

        Changes
        -------

        Added the new property to hive-default.xml

        Summary
        -------

        I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double.

        I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode.

        This addresses bug HIVE-2427.
        https://issues.apache.org/jira/browse/HIVE-2427

        Diffs (updated)


        trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293
        trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293
        trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293
        trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293
        trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293
        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293
        trunk/conf/hive-default.xml 1164293

        Diff: https://reviews.apache.org/r/1705/diff

        Testing
        -------

        I added several more tests to demonstrate the errors.

        I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1705/ ----------------------------------------------------------- (Updated 2011-09-06 17:41:24.308161) Review request for hive and Siying Dong. Changes ------- Added the new property to hive-default.xml Summary ------- I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double. I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode. This addresses bug HIVE-2427 . https://issues.apache.org/jira/browse/HIVE-2427 Diffs (updated) trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293 trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293 trunk/conf/hive-default.xml 1164293 Diff: https://reviews.apache.org/r/1705/diff Testing ------- I added several more tests to demonstrate the errors. I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1705/
        -----------------------------------------------------------

        (Updated 2011-09-07 01:33:36.605379)

        Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong.

        Summary
        -------

        I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double.

        I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode.

        This addresses bug HIVE-2427.
        https://issues.apache.org/jira/browse/HIVE-2427

        Diffs


        trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293
        trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293
        trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293
        trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293
        trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293
        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293
        trunk/conf/hive-default.xml 1164293

        Diff: https://reviews.apache.org/r/1705/diff

        Testing
        -------

        I added several more tests to demonstrate the errors.

        I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1705/ ----------------------------------------------------------- (Updated 2011-09-07 01:33:36.605379) Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. Summary ------- I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double. I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode. This addresses bug HIVE-2427 . https://issues.apache.org/jira/browse/HIVE-2427 Diffs trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293 trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293 trunk/conf/hive-default.xml 1164293 Diff: https://reviews.apache.org/r/1705/diff Testing ------- I added several more tests to demonstrate the errors. I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output. Thanks, Kevin
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1705/
        -----------------------------------------------------------

        (Updated 2011-11-04 19:48:37.718693)

        Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong.

        Changes
        -------

        It's been a while, so I ran svn up and fixed the conflicts.

        I ran the tests and verified they passed.

        Summary
        -------

        I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double.

        I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode.

        This addresses bug HIVE-2427.
        https://issues.apache.org/jira/browse/HIVE-2427

        Diffs (updated)


        trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1197375
        trunk/conf/hive-default.xml 1197375
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1197375
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1197375
        trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1197375
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1197375
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1197375
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1197375
        trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1197375
        trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1197375
        trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1197375

        Diff: https://reviews.apache.org/r/1705/diff

        Testing
        -------

        I added several more tests to demonstrate the errors.

        I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output.

        Thanks,

        Kevin

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1705/ ----------------------------------------------------------- (Updated 2011-11-04 19:48:37.718693) Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. Changes ------- It's been a while, so I ran svn up and fixed the conflicts. I ran the tests and verified they passed. Summary ------- I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double. I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode. This addresses bug HIVE-2427 . https://issues.apache.org/jira/browse/HIVE-2427 Diffs (updated) trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1197375 trunk/conf/hive-default.xml 1197375 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1197375 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1197375 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1197375 trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1197375 trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1197375 trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1197375 trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1197375 trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1197375 trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1197375 Diff: https://reviews.apache.org/r/1705/diff Testing ------- I added several more tests to demonstrate the errors. I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output. Thanks, Kevin
        Hide
        Phabricator added a comment -

        kevinwilfong requested code review of "HIVE-2427 [jira] Warn user that precision is lost when bigint is implicitly cast to double in joins and unions.".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2427

        (Ran svn up to avoid merge conflicts and moved review to Phabricator)

        I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double.

        I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode.

        When a bigint is implicitly cast to a double (when a bigint is involved in an equality expression with a string or double as part of a join condition, or when a bigint column is to be combined with a string or double column as part of a union) precision may be lost, resulting in unexpected behavior. Until we fix the underlying issue we should throw an error in strict mode, and a warning in nonstrict mode alerting the user about this.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D2181

        AFFECTED FILES
        conf/hive-default.xml.template
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/compare_string_bigint.q.out
        ql/src/test/results/clientnegative/compare_string_bigint_union.q.out
        ql/src/test/results/clientnegative/compare_string_bigint_join.q.out
        ql/src/test/results/clientnegative/compare_double_bigint_union.q.out
        ql/src/test/results/clientnegative/compare_double_bigint_join.q.out
        ql/src/test/results/clientnegative/compare_double_bigint.q.out
        ql/src/test/results/clientpositive/filter_join_breaktask2.q.out
        ql/src/test/queries/clientnegative/compare_double_bigint_union.q
        ql/src/test/queries/clientnegative/compare_string_bigint_join.q
        ql/src/test/queries/clientnegative/compare_double_bigint.q
        ql/src/test/queries/clientnegative/compare_double_bigint_join.q
        ql/src/test/queries/clientnegative/compare_string_bigint.q
        ql/src/test/queries/clientnegative/compare_string_bigint_union.q
        ql/src/test/queries/clientpositive/filter_join_breaktask2.q
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/4779/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - kevinwilfong requested code review of " HIVE-2427 [jira] Warn user that precision is lost when bigint is implicitly cast to double in joins and unions.". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2427 (Ran svn up to avoid merge conflicts and moved review to Phabricator) I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double. I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode. When a bigint is implicitly cast to a double (when a bigint is involved in an equality expression with a string or double as part of a join condition, or when a bigint column is to be combined with a string or double column as part of a union) precision may be lost, resulting in unexpected behavior. Until we fix the underlying issue we should throw an error in strict mode, and a warning in nonstrict mode alerting the user about this. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2181 AFFECTED FILES conf/hive-default.xml.template common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/compare_string_bigint.q.out ql/src/test/results/clientnegative/compare_string_bigint_union.q.out ql/src/test/results/clientnegative/compare_string_bigint_join.q.out ql/src/test/results/clientnegative/compare_double_bigint_union.q.out ql/src/test/results/clientnegative/compare_double_bigint_join.q.out ql/src/test/results/clientnegative/compare_double_bigint.q.out ql/src/test/results/clientpositive/filter_join_breaktask2.q.out ql/src/test/queries/clientnegative/compare_double_bigint_union.q ql/src/test/queries/clientnegative/compare_string_bigint_join.q ql/src/test/queries/clientnegative/compare_double_bigint.q ql/src/test/queries/clientnegative/compare_double_bigint_join.q ql/src/test/queries/clientnegative/compare_string_bigint.q ql/src/test/queries/clientnegative/compare_string_bigint_union.q ql/src/test/queries/clientpositive/filter_join_breaktask2.q ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4779/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-2427 [jira] Warn user that precision is lost when bigint is implicitly cast to double in joins and unions.".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:5180 This piece of code seems to be repeated - can you make a function
        out of it, and reuse it.
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:198 same as above

        REVISION DETAIL
        https://reviews.facebook.net/D2181

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-2427 [jira] Warn user that precision is lost when bigint is implicitly cast to double in joins and unions.". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:5180 This piece of code seems to be repeated - can you make a function out of it, and reuse it. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:198 same as above REVISION DETAIL https://reviews.facebook.net/D2181

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development