Hive
  1. Hive
  2. HIVE-5376

Hive does not honor type for partition columns when altering column type

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: CLI
    • Labels:
      None

      Description

      Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change.
      alter_partition_coltype.q (or some other test?) checks this behavior right now.

      1. HIVE-5376.1.patch
        68 kB
        Hari Sankar Sivarama Subramaniyan
      2. HIVE-5376.2.patch
        86 kB
        Hari Sankar Sivarama Subramaniyan
      3. HIVE-5376.3.patch
        90 kB
        Hari Sankar Sivarama Subramaniyan

        Issue Links

          Activity

          Hide
          Chaoyu Tang added a comment -

          Thanks Sergey Shelukhin for the clarification.

          Show
          Chaoyu Tang added a comment - Thanks Sergey Shelukhin for the clarification.
          Hide
          Sergey Shelukhin added a comment -

          I think whatever that JIRA was doing needs some code from ql module. If it doesn't then the below is irrelevant. If it does, the problem is that ql module depends on metastore module, so the reverse dependency cannot be added at compile time. So, if you look at Expr stuff, it has a proxy class where interface is in metastore, impl is in ql, and it's instantiated in metastore once via reflection. Or someone could unf*** Hive module structure, but that seems like too big of a task

          Show
          Sergey Shelukhin added a comment - I think whatever that JIRA was doing needs some code from ql module. If it doesn't then the below is irrelevant. If it does, the problem is that ql module depends on metastore module, so the reverse dependency cannot be added at compile time. So, if you look at Expr stuff, it has a proxy class where interface is in metastore, impl is in ql, and it's instantiated in metastore once via reflection. Or someone could unf*** Hive module structure, but that seems like too big of a task
          Hide
          Chaoyu Tang added a comment -

          Sergey Shelukhin & Hari Sankar Sivarama Subramaniyan Could you elaborate the comment on
          https://issues.apache.org/jira/browse/HIVE-5376?focusedCommentId=13960805&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13960805
          I do not quite understand what the difficulty is to implement this JIRA at metastore site and why metastore needs QL.
          Thanks.

          Show
          Chaoyu Tang added a comment - Sergey Shelukhin & Hari Sankar Sivarama Subramaniyan Could you elaborate the comment on https://issues.apache.org/jira/browse/HIVE-5376?focusedCommentId=13960805&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13960805 I do not quite understand what the difficulty is to implement this JIRA at metastore site and why metastore needs QL. Thanks.
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Harish Butani added a comment -

          thanks Hari, Sergey

          Show
          Harish Butani added a comment - thanks Hari, Sergey
          Hide
          Lefty Leverenz added a comment -

          Filed HIVE-6943 to handle the Constant Expression issue.

          ... but you meant to type HIVE-6934.

          Show
          Lefty Leverenz added a comment - Filed HIVE-6943 to handle the Constant Expression issue. ... but you meant to type HIVE-6934 .
          Hide
          Harish Butani added a comment -

          +1
          Filed HIVE-6943 to handle the Constant Expression issue.

          Show
          Harish Butani added a comment - +1 Filed HIVE-6943 to handle the Constant Expression issue.
          Hide
          Hari Sankar Sivarama Subramaniyan added a comment -

          Harish Butani The DEFAULT_PARTITION issue is no longer present. It looks like a local setup issue. Please look at the new patch

          Show
          Hari Sankar Sivarama Subramaniyan added a comment - Harish Butani The DEFAULT_PARTITION issue is no longer present. It looks like a local setup issue. Please look at the new patch
          Hide
          Harish Butani added a comment -

          Removed this from 0.13 list.
          Hari Sankar Sivarama Subramaniyan please look into the failure when there is a DEFAULT_PARTITION

          Show
          Harish Butani added a comment - Removed this from 0.13 list. Hari Sankar Sivarama Subramaniyan please look into the failure when there is a DEFAULT_PARTITION
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12639448/HIVE-5376.2.patch

          ERROR: -1 due to 4 failed/errored test(s), 5571 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_leftsemijoin_mr
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_partition_coltype_invalidtype_conversion
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2200/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2200/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12639448

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12639448/HIVE-5376.2.patch ERROR: -1 due to 4 failed/errored test(s), 5571 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_leftsemijoin_mr org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_partition_coltype_invalidtype_conversion Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2200/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2200/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12639448
          Hide
          Sergey Shelukhin added a comment -

          +1

          Show
          Sergey Shelukhin added a comment - +1
          Hide
          Sergey Shelukhin added a comment -

          Discussed w/Harish. It should be ok to do on client now.
          It's better to do close to data (metadata ) but it will be difficult to achieve because metastore cannot depend on QL during compilation so some proxy class will need to be used, like for getPartitionsByExpr.
          Maybe another jira can be filed...

          Show
          Sergey Shelukhin added a comment - Discussed w/Harish. It should be ok to do on client now. It's better to do close to data (metadata ) but it will be difficult to achieve because metastore cannot depend on QL during compilation so some proxy class will need to be used, like for getPartitionsByExpr. Maybe another jira can be filed...
          Hide
          Sergey Shelukhin added a comment -

          Hmm... would it be be better to do on metastore side? Both safer, cause someone else like HCat can all the API, and more efficient to get data

          Show
          Sergey Shelukhin added a comment - Hmm... would it be be better to do on metastore side? Both safer, cause someone else like HCat can all the API, and more efficient to get data
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12638360/HIVE-5376.1.patch

          SUCCESS: +1 5543 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2100/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2100/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12638360

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12638360/HIVE-5376.1.patch SUCCESS: +1 5543 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2100/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2100/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12638360
          Show
          Hari Sankar Sivarama Subramaniyan added a comment - https://reviews.apache.org/r/19964/
          Hide
          Hari Sankar Sivarama Subramaniyan added a comment -

          cc-ing Harish Butani for review.

          Show
          Hari Sankar Sivarama Subramaniyan added a comment - cc-ing Harish Butani for review.
          Hide
          Vikram Dixit K added a comment -

          Hari Sankar Sivarama Subramaniyan I am not currently working on it. Please go ahead and assign it to yourself if you are working on it.

          Show
          Vikram Dixit K added a comment - Hari Sankar Sivarama Subramaniyan I am not currently working on it. Please go ahead and assign it to yourself if you are working on it.
          Hide
          Hari Sankar Sivarama Subramaniyan added a comment -

          Thanks Sergey Shelukhin for marking it a duplicate of HIVE-6775. Vikram Dixit Kplease let me know if you are working on this.

          Show
          Hari Sankar Sivarama Subramaniyan added a comment - Thanks Sergey Shelukhin for marking it a duplicate of HIVE-6775 . Vikram Dixit K please let me know if you are working on this.
          Hide
          Sergey Shelukhin added a comment -
          Show
          Sergey Shelukhin added a comment - Vikram Dixit K filed

            People

            • Assignee:
              Hari Sankar Sivarama Subramaniyan
              Reporter:
              Sergey Shelukhin
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development