Hive
  1. Hive
  2. HIVE-5376

Hive does not honor type for partition columns when altering column type

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: CLI
    • Labels:
      None

      Description

      Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change.
      alter_partition_coltype.q (or some other test?) checks this behavior right now.

      1. HIVE-5376.1.patch
        68 kB
        Hari Sankar Sivarama Subramaniyan
      2. HIVE-5376.2.patch
        86 kB
        Hari Sankar Sivarama Subramaniyan
      3. HIVE-5376.3.patch
        90 kB
        Hari Sankar Sivarama Subramaniyan

        Issue Links

          Activity

          Sergey Shelukhin created issue -
          Hide
          Sergey Shelukhin added a comment -
          Show
          Sergey Shelukhin added a comment - Vikram Dixit K filed
          Sergey Shelukhin made changes -
          Field Original Value New Value
          Description Followup for HBASE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change.
          alter_partition_coltype.q (or some other test?) checks this behavior right now.
          Followup for HIVE-5297. If partition column of type string is changed to int, the data is not verified. The values for partition columns are all in metastore db, so it's easy to check and fail the type change.
          alter_partition_coltype.q (or some other test?) checks this behavior right now.
          Vikram Dixit K made changes -
          Assignee Vikram Dixit K [ vikram.dixit ]
          Sergey Shelukhin made changes -
          Link This issue is duplicated by HIVE-6775 [ HIVE-6775 ]
          Hide
          Hari Sankar Sivarama Subramaniyan added a comment -

          Thanks Sergey Shelukhin for marking it a duplicate of HIVE-6775. Vikram Dixit Kplease let me know if you are working on this.

          Show
          Hari Sankar Sivarama Subramaniyan added a comment - Thanks Sergey Shelukhin for marking it a duplicate of HIVE-6775 . Vikram Dixit K please let me know if you are working on this.
          Hide
          Vikram Dixit K added a comment -

          Hari Sankar Sivarama Subramaniyan I am not currently working on it. Please go ahead and assign it to yourself if you are working on it.

          Show
          Vikram Dixit K added a comment - Hari Sankar Sivarama Subramaniyan I am not currently working on it. Please go ahead and assign it to yourself if you are working on it.
          Hari Sankar Sivarama Subramaniyan made changes -
          Assignee Vikram Dixit K [ vikram.dixit ] Hari Sankar Sivarama Subramaniyan [ hsubramaniyan ]
          Hide
          Hari Sankar Sivarama Subramaniyan added a comment -

          cc-ing Harish Butani for review.

          Show
          Hari Sankar Sivarama Subramaniyan added a comment - cc-ing Harish Butani for review.
          Hari Sankar Sivarama Subramaniyan made changes -
          Attachment HIVE-5376.1.patch [ 12638360 ]
          Hari Sankar Sivarama Subramaniyan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Show
          Hari Sankar Sivarama Subramaniyan added a comment - https://reviews.apache.org/r/19964/
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12638360/HIVE-5376.1.patch

          SUCCESS: +1 5543 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2100/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2100/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12638360

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12638360/HIVE-5376.1.patch SUCCESS: +1 5543 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2100/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2100/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12638360
          Hide
          Sergey Shelukhin added a comment -

          Hmm... would it be be better to do on metastore side? Both safer, cause someone else like HCat can all the API, and more efficient to get data

          Show
          Sergey Shelukhin added a comment - Hmm... would it be be better to do on metastore side? Both safer, cause someone else like HCat can all the API, and more efficient to get data
          Hide
          Sergey Shelukhin added a comment -

          Discussed w/Harish. It should be ok to do on client now.
          It's better to do close to data (metadata ) but it will be difficult to achieve because metastore cannot depend on QL during compilation so some proxy class will need to be used, like for getPartitionsByExpr.
          Maybe another jira can be filed...

          Show
          Sergey Shelukhin added a comment - Discussed w/Harish. It should be ok to do on client now. It's better to do close to data (metadata ) but it will be difficult to achieve because metastore cannot depend on QL during compilation so some proxy class will need to be used, like for getPartitionsByExpr. Maybe another jira can be filed...
          Hide
          Sergey Shelukhin added a comment -

          +1

          Show
          Sergey Shelukhin added a comment - +1
          Hari Sankar Sivarama Subramaniyan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hari Sankar Sivarama Subramaniyan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hari Sankar Sivarama Subramaniyan made changes -
          Attachment HIVE-5376.2.patch [ 12639448 ]
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12639448/HIVE-5376.2.patch

          ERROR: -1 due to 4 failed/errored test(s), 5571 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_leftsemijoin_mr
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_partition_coltype_invalidtype_conversion
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2200/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2200/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12639448

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12639448/HIVE-5376.2.patch ERROR: -1 due to 4 failed/errored test(s), 5571 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_leftsemijoin_mr org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_partition_coltype_invalidtype_conversion Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2200/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2200/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12639448
          Hide
          Harish Butani added a comment -

          Removed this from 0.13 list.
          Hari Sankar Sivarama Subramaniyan please look into the failure when there is a DEFAULT_PARTITION

          Show
          Harish Butani added a comment - Removed this from 0.13 list. Hari Sankar Sivarama Subramaniyan please look into the failure when there is a DEFAULT_PARTITION
          Hari Sankar Sivarama Subramaniyan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hari Sankar Sivarama Subramaniyan made changes -
          Attachment HIVE-5376.3.patch [ 12640531 ]
          Hide
          Hari Sankar Sivarama Subramaniyan added a comment -

          Harish Butani The DEFAULT_PARTITION issue is no longer present. It looks like a local setup issue. Please look at the new patch

          Show
          Hari Sankar Sivarama Subramaniyan added a comment - Harish Butani The DEFAULT_PARTITION issue is no longer present. It looks like a local setup issue. Please look at the new patch
          Hari Sankar Sivarama Subramaniyan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Harish Butani added a comment -

          +1
          Filed HIVE-6943 to handle the Constant Expression issue.

          Show
          Harish Butani added a comment - +1 Filed HIVE-6943 to handle the Constant Expression issue.
          Lefty Leverenz made changes -
          Link This issue is related to HIVE-6934 [ HIVE-6934 ]
          Hide
          Lefty Leverenz added a comment -

          Filed HIVE-6943 to handle the Constant Expression issue.

          ... but you meant to type HIVE-6934.

          Show
          Lefty Leverenz added a comment - Filed HIVE-6943 to handle the Constant Expression issue. ... but you meant to type HIVE-6934 .
          Hari Sankar Sivarama Subramaniyan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hari Sankar Sivarama Subramaniyan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Harish Butani added a comment -

          thanks Hari, Sergey

          Show
          Harish Butani added a comment - thanks Hari, Sergey
          Harish Butani made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Ashutosh Chauhan made changes -
          Fix Version/s 0.14.0 [ 12326450 ]
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Thejas M Nair made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Chaoyu Tang added a comment -

          Sergey Shelukhin & Hari Sankar Sivarama Subramaniyan Could you elaborate the comment on
          https://issues.apache.org/jira/browse/HIVE-5376?focusedCommentId=13960805&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13960805
          I do not quite understand what the difficulty is to implement this JIRA at metastore site and why metastore needs QL.
          Thanks.

          Show
          Chaoyu Tang added a comment - Sergey Shelukhin & Hari Sankar Sivarama Subramaniyan Could you elaborate the comment on https://issues.apache.org/jira/browse/HIVE-5376?focusedCommentId=13960805&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13960805 I do not quite understand what the difficulty is to implement this JIRA at metastore site and why metastore needs QL. Thanks.
          Hide
          Sergey Shelukhin added a comment -

          I think whatever that JIRA was doing needs some code from ql module. If it doesn't then the below is irrelevant. If it does, the problem is that ql module depends on metastore module, so the reverse dependency cannot be added at compile time. So, if you look at Expr stuff, it has a proxy class where interface is in metastore, impl is in ql, and it's instantiated in metastore once via reflection. Or someone could unf*** Hive module structure, but that seems like too big of a task

          Show
          Sergey Shelukhin added a comment - I think whatever that JIRA was doing needs some code from ql module. If it doesn't then the below is irrelevant. If it does, the problem is that ql module depends on metastore module, so the reverse dependency cannot be added at compile time. So, if you look at Expr stuff, it has a proxy class where interface is in metastore, impl is in ql, and it's instantiated in metastore once via reflection. Or someone could unf*** Hive module structure, but that seems like too big of a task
          Hide
          Chaoyu Tang added a comment -

          Thanks Sergey Shelukhin for the clarification.

          Show
          Chaoyu Tang added a comment - Thanks Sergey Shelukhin for the clarification.
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          19d 20h 26m 3 Hari Sankar Sivarama Subramaniyan 22/Apr/14 18:56
          Open Open Patch Available Patch Available
          188d 24m 4 Hari Sankar Sivarama Subramaniyan 22/Apr/14 18:56
          Patch Available Patch Available Resolved Resolved
          11m 18s 1 Harish Butani 22/Apr/14 19:07
          Resolved Resolved Closed Closed
          205d 36m 1 Thejas M Nair 13/Nov/14 19:44

            People

            • Assignee:
              Hari Sankar Sivarama Subramaniyan
              Reporter:
              Sergey Shelukhin
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development