Hive
  1. Hive
  2. HIVE-6945

issues with dropping partitions on Oracle

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.14.0, 0.13.1
    • Component/s: None
    • Labels:
      None

      Description

      1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is translated by DN into decimal rather than long. This appears to be specific to some cases because it seemed to have worked before (different version of Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
      2) When partition dropping code falls back to JDO, it creates objects to return, then drops partitions. It appears that dropping makes DN objects invalid. We create metastore partition objects out of DN objects before drop, however the list of partition column values is re-used, rather than copied, into these. DN appears to clear this list during drop, so the returned object becomes invalid and the exception is thrown.

      1. HIVE-6945.01.patch
        12 kB
        Sergey Shelukhin
      2. HIVE-6945.02.patch
        12 kB
        Sergey Shelukhin
      3. HIVE-6945.patch
        9 kB
        Sergey Shelukhin
      4. HIVE-6945-0.13.1.patch
        20 kB
        Sushanth Sowmyan

        Activity

        Hide
        Xuefu Zhang added a comment -

        Could we have some description about what issues are in focus here? The title alone doesn't seem providing any essential information that help the readers.

        Show
        Xuefu Zhang added a comment - Could we have some description about what issues are in focus here? The title alone doesn't seem providing any essential information that help the readers.
        Hide
        Sergey Shelukhin added a comment -

        Updated

        Show
        Sergey Shelukhin added a comment - Updated
        Show
        Sergey Shelukhin added a comment - https://reviews.apache.org/r/20565/
        Hide
        Sergey Shelukhin added a comment -

        some missed places on sql path

        Show
        Sergey Shelukhin added a comment - some missed places on sql path
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12641396/HIVE-6945.02.patch

        ERROR: -1 due to 48 failed/errored test(s), 5417 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter_partitioned
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
        org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/10/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/10/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 48 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12641396

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12641396/HIVE-6945.02.patch ERROR: -1 due to 48 failed/errored test(s), 5417 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter_partitioned org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/10/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/10/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 48 tests failed This message is automatically generated. ATTACHMENT ID: 12641396
        Hide
        Sergey Shelukhin added a comment -

        Most of these tests pass on my machine (others are running) and the output looks bogus (has wrong change list, and no meaningful test results). I think this may be precommit test glitch.

        Show
        Sergey Shelukhin added a comment - Most of these tests pass on my machine (others are running) and the output looks bogus (has wrong change list, and no meaningful test results). I think this may be precommit test glitch.
        Hide
        Ashutosh Chauhan added a comment -

        +1

        Show
        Ashutosh Chauhan added a comment - +1
        Hide
        Sergey Shelukhin added a comment -

        committed to trunk

        Show
        Sergey Shelukhin added a comment - committed to trunk
        Hide
        Sushanth Sowmyan added a comment -

        FYI, on testing 0.13.0 RC0, this patch seemed to be the one git bisect identifies as causing a couple of test failures.

        The test failures are as follows:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property

        The logs for the git bisect run for this bug are available over at http://people.apache.org/~khorgath/releases/0.13.1_RC0/test_failures/HIVE-6945.bisect/

        The .sh files there were the scripts used to test hive for the bugs in question, and the file prior.patch available there is the patch that you will need to apply on top of 0.13.0 if you want to arrive at the last clean state before this patch was introduced and the issue was observed.

        Show
        Sushanth Sowmyan added a comment - FYI, on testing 0.13.0 RC0, this patch seemed to be the one git bisect identifies as causing a couple of test failures. The test failures are as follows: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property The logs for the git bisect run for this bug are available over at http://people.apache.org/~khorgath/releases/0.13.1_RC0/test_failures/HIVE-6945.bisect/ The .sh files there were the scripts used to test hive for the bugs in question, and the file prior.patch available there is the patch that you will need to apply on top of 0.13.0 if you want to arrive at the last clean state before this patch was introduced and the issue was observed.
        Hide
        Sushanth Sowmyan added a comment -

        Attaching updated patch for 0.13.1 that causes all tests to pass. (It was to do with reordering of TBLPROPERTIES contents in the .out files)

        Show
        Sushanth Sowmyan added a comment - Attaching updated patch for 0.13.1 that causes all tests to pass. (It was to do with reordering of TBLPROPERTIES contents in the .out files)
        Hide
        Thejas M Nair added a comment -

        This jira has been fixed as part of 0.13.1 release. If you find further issues, please create a new jira and link it to this one.

        Show
        Thejas M Nair added a comment - This jira has been fixed as part of 0.13.1 release. If you find further issues, please create a new jira and link it to this one.

          People

          • Assignee:
            Sergey Shelukhin
            Reporter:
            Sergey Shelukhin
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development