Hive
  1. Hive
  2. HIVE-2813

Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.8.1, 0.9.0
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
    • Tags:
      hive, sematic analyzer

      Description

      Oftentimes people try to write queries like
      SELECT *
      FROM table
      WHERE ds = 2011-08-03

      This won't work because quotes are missing, and it'll actually try to filter on ds = "2000".

      People run into this pretty regularly; a simple check on whether the types match exactly in partition predicates would make this a lot less likely to happen.

        Activity

        Show
        Dmitry Soshnikov added a comment - https://reviews.facebook.net/D1803
        Hide
        Phabricator added a comment -

        dmitrys updated the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".
        Reviewers: kevinwilfong, JIRA

        missed with a task number

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
        ql/src/test/queries/clientnegative/select_partcol_diff_types.q
        ql/src/test/results/clientnegative/select_partcol_diff_types.q.out

        Show
        Phabricator added a comment - dmitrys updated the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". Reviewers: kevinwilfong, JIRA missed with a task number REVISION DETAIL https://reviews.facebook.net/D1803 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java ql/src/test/queries/clientnegative/select_partcol_diff_types.q ql/src/test/results/clientnegative/select_partcol_diff_types.q.out
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java:58 This should mention the variable that needs to be set to strict/nonstrict to turn this error on/off, and should mention it only applies to comparisons where one side is a partition column.
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:223-230 If the child is a column you should make sure that child is a partition column.
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:234 It might be better to add a new variable to the HiveConf.ConfVars, something like PARTITION_COMPARISON_MODE, which only controls whether or not an error is thrown here or not.

        We have too many things which are controlled by HIVEMAPREDMODE, setting it to nonstrict for this could turn on other things the user doesn't want.

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java:58 This should mention the variable that needs to be set to strict/nonstrict to turn this error on/off, and should mention it only applies to comparisons where one side is a partition column. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:223-230 If the child is a column you should make sure that child is a partition column. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:234 It might be better to add a new variable to the HiveConf.ConfVars, something like PARTITION_COMPARISON_MODE, which only controls whether or not an error is thrown here or not. We have too many things which are controlled by HIVEMAPREDMODE, setting it to nonstrict for this could turn on other things the user doesn't want. REVISION DETAIL https://reviews.facebook.net/D1803
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:235-239 It would be good to mention here as well that this only applies to partition columns.

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:235-239 It would be good to mention here as well that this only applies to partition columns. REVISION DETAIL https://reviews.facebook.net/D1803
        Hide
        Phabricator added a comment -

        dmitrys updated the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".
        Reviewers: kevinwilfong, JIRA, njain

        Yes, good points; fixed

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
        ql/src/test/queries/clientnegative/select_partcol_diff_types.q
        ql/src/test/results/clientnegative/select_partcol_diff_types.q.out

        Show
        Phabricator added a comment - dmitrys updated the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". Reviewers: kevinwilfong, JIRA, njain Yes, good points; fixed REVISION DETAIL https://reviews.facebook.net/D1803 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java ql/src/test/queries/clientnegative/select_partcol_diff_types.q ql/src/test/results/clientnegative/select_partcol_diff_types.q.out
        Hide
        Phabricator added a comment -

        cwsteinbach has requested changes to the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".

        INLINE COMMENTS
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:342 Please add this to conf/hive-default.xml.template along with a description of what it does, and the allowed values.
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java:59 spelling: "nonstric"
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:215 spelling
        ql/src/test/queries/clientnegative/select_partcol_diff_types.q:1 Please add a positive testcase that explicitly sets hive.partition.comparison.mode=nonstrict so that we can test the WARNING message.

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        BRANCH
        trunk

        Show
        Phabricator added a comment - cwsteinbach has requested changes to the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:342 Please add this to conf/hive-default.xml.template along with a description of what it does, and the allowed values. ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java:59 spelling: "nonstric" ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:215 spelling ql/src/test/queries/clientnegative/select_partcol_diff_types.q:1 Please add a positive testcase that explicitly sets hive.partition.comparison.mode=nonstrict so that we can test the WARNING message. REVISION DETAIL https://reviews.facebook.net/D1803 BRANCH trunk
        Hide
        Phabricator added a comment -

        cwsteinbach has commented on the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".

        INLINE COMMENTS
        ql/src/test/results/clientnegative/select_partcol_diff_types.q.out:1 Looks like there's actually another bug here. Shouldn't the argument be 2012-2-17 = 1993?

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        BRANCH
        trunk

        Show
        Phabricator added a comment - cwsteinbach has commented on the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS ql/src/test/results/clientnegative/select_partcol_diff_types.q.out:1 Looks like there's actually another bug here. Shouldn't the argument be 2012-2-17 = 1993? REVISION DETAIL https://reviews.facebook.net/D1803 BRANCH trunk
        Hide
        Phabricator added a comment -

        dmitrys updated the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".
        Reviewers: kevinwilfong, JIRA, njain, cwsteinbach

        Yep, thanks; fixed

        And yes, perhaps it's another bug with reducing the 'date' to int; not sure since don't know the full specific (I thought '17' is error code of
        argument, not the reduced value).

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        conf/hive-default.xml.template
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
        ql/src/test/queries/clientnegative/select_partcol_diff_types.q
        ql/src/test/queries/clientpositive/select_partcol_diff_types.q
        ql/src/test/results/clientnegative/select_partcol_diff_types.q.out
        ql/src/test/results/clientpositive/select_partcol_diff_types.q.out

        Show
        Phabricator added a comment - dmitrys updated the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". Reviewers: kevinwilfong, JIRA, njain, cwsteinbach Yep, thanks; fixed And yes, perhaps it's another bug with reducing the 'date' to int; not sure since don't know the full specific (I thought '17' is error code of argument, not the reduced value). REVISION DETAIL https://reviews.facebook.net/D1803 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java ql/src/test/queries/clientnegative/select_partcol_diff_types.q ql/src/test/queries/clientpositive/select_partcol_diff_types.q ql/src/test/results/clientnegative/select_partcol_diff_types.q.out ql/src/test/results/clientpositive/select_partcol_diff_types.q.out
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".

        INLINE COMMENTS
        ql/src/test/results/clientnegative/select_partcol_diff_types.q.out:1 Looks like this is a bug in the getText method of ErrorMsg, when it uses the rightmost node in the AST to construct the error message String which in this case is 17.

        I filed a JIRA for this here https://issues.apache.org/jira/browse/HIVE-2830

        In the mean time, Dmitry, using ds = 2012, for example, should still produce the results you're looking for and avoid the bug.

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS ql/src/test/results/clientnegative/select_partcol_diff_types.q.out:1 Looks like this is a bug in the getText method of ErrorMsg, when it uses the rightmost node in the AST to construct the error message String which in this case is 17. I filed a JIRA for this here https://issues.apache.org/jira/browse/HIVE-2830 In the mean time, Dmitry, using ds = 2012, for example, should still produce the results you're looking for and avoid the bug. REVISION DETAIL https://reviews.facebook.net/D1803
        Hide
        Phabricator added a comment -

        dmitrys updated the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".
        Reviewers: kevinwilfong, JIRA, njain, cwsteinbach

        OK, used 2012 as an example

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        conf/hive-default.xml.template
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
        ql/src/test/queries/clientnegative/select_partcol_diff_types.q
        ql/src/test/queries/clientpositive/select_partcol_diff_types.q
        ql/src/test/results/clientnegative/select_partcol_diff_types.q.out
        ql/src/test/results/clientpositive/select_partcol_diff_types.q.out

        Show
        Phabricator added a comment - dmitrys updated the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". Reviewers: kevinwilfong, JIRA, njain, cwsteinbach OK, used 2012 as an example REVISION DETAIL https://reviews.facebook.net/D1803 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java ql/src/test/queries/clientnegative/select_partcol_diff_types.q ql/src/test/queries/clientpositive/select_partcol_diff_types.q ql/src/test/results/clientnegative/select_partcol_diff_types.q.out ql/src/test/results/clientpositive/select_partcol_diff_types.q.out
        Hide
        Phabricator added a comment -

        kevinwilfong has accepted the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".

        +1 Looks good to me now.

        @Carl
        It looks like he's addressed all of your concerns. I'll commit on Monday unless you see any other issues.

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        BRANCH
        trunk

        Show
        Phabricator added a comment - kevinwilfong has accepted the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". +1 Looks good to me now. @Carl It looks like he's addressed all of your concerns. I'll commit on Monday unless you see any other issues. REVISION DETAIL https://reviews.facebook.net/D1803 BRANCH trunk
        Hide
        Phabricator added a comment -

        kevinwilfong has requested changes to the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".

        The patch is out of date, could you run svn up.

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        BRANCH
        trunk

        Show
        Phabricator added a comment - kevinwilfong has requested changes to the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". The patch is out of date, could you run svn up. REVISION DETAIL https://reviews.facebook.net/D1803 BRANCH trunk
        Hide
        Phabricator added a comment -

        dmitrys updated the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".
        Reviewers: kevinwilfong, JIRA, njain, cwsteinbach

        Updated

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        AFFECTED FILES
        conf/hive-default.xml.template
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientpositive/select_partcol_diff_types.q.out
        ql/src/test/results/clientnegative/select_partcol_diff_types.q.out
        ql/src/test/queries/clientpositive/select_partcol_diff_types.q
        ql/src/test/queries/clientnegative/select_partcol_diff_types.q
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java

        Show
        Phabricator added a comment - dmitrys updated the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". Reviewers: kevinwilfong, JIRA, njain, cwsteinbach Updated REVISION DETAIL https://reviews.facebook.net/D1803 AFFECTED FILES conf/hive-default.xml.template common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientpositive/select_partcol_diff_types.q.out ql/src/test/results/clientnegative/select_partcol_diff_types.q.out ql/src/test/queries/clientpositive/select_partcol_diff_types.q ql/src/test/queries/clientnegative/select_partcol_diff_types.q ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        Hide
        Phabricator added a comment -

        kevinwilfong has accepted the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.".

        Thanks, running tests.

        REVISION DETAIL
        https://reviews.facebook.net/D1803

        BRANCH
        svn

        Show
        Phabricator added a comment - kevinwilfong has accepted the revision " HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". Thanks, running tests. REVISION DETAIL https://reviews.facebook.net/D1803 BRANCH svn
        Hide
        Ashutosh Chauhan added a comment -

        Unlinking from 0.9

        Show
        Ashutosh Chauhan added a comment - Unlinking from 0.9

          People

          • Assignee:
            Unassigned
            Reporter:
            Dmitry Soshnikov
          • Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development