Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17523

Insert into druid table hangs Hive server2 in an infinite loop

Details

    Description

      Inserting data via insert into table backed by druid can lead to a Hive server hang.
      This is due to some bug in the naming of druid segments partitions.
      To reproduce the issue

      drop table login_hive;
      create table login_hive(`timecolumn` timestamp, `userid` string, `num_l` double);
      insert into login_hive values ('2015-01-01 00:00:00', 'user1', 5);
      insert into login_hive values ('2015-01-01 01:00:00', 'user2', 4);
      insert into login_hive values ('2015-01-01 02:00:00', 'user3', 2);
      
      insert into login_hive values ('2015-01-02 00:00:00', 'user1', 1);
      insert into login_hive values ('2015-01-02 01:00:00', 'user2', 2);
      insert into login_hive values ('2015-01-02 02:00:00', 'user3', 8);
      
      insert into login_hive values ('2015-01-03 00:00:00', 'user1', 5);
      insert into login_hive values ('2015-01-03 01:00:00', 'user2', 9);
      insert into login_hive values ('2015-01-03 04:00:00', 'user3', 2);
      
      insert into login_hive values ('2015-03-09 00:00:00', 'user3', 5);
      insert into login_hive values ('2015-03-09 01:00:00', 'user1', 0);
      insert into login_hive values ('2015-03-09 05:00:00', 'user2', 0);
      
      
      drop table login_druid;
      CREATE TABLE login_druid
      STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
      TBLPROPERTIES ("druid.datasource" = "druid_login_test_tmp", "druid.segment.granularity" = "DAY", "druid.query.granularity" = "HOUR")
      AS
      select `timecolumn` as `__time`, `userid`, `num_l` FROM login_hive;
      select * FROM login_druid;
      
      insert into login_druid values ('2015-03-09 05:00:00', 'user4', 0); 
      

      This patch unifies the logic of pushing and segments naming by using Druid data segment pusher as much as possible.
      This patch also has some minor code refactoring and test enhancements.

      Attachments

        1. HIVE-17523.patch
          54 kB
          Slim Bouguerra
        2. HIVE-17523.2.patch
          53 kB
          Slim Bouguerra

        Issue Links

          Activity

            bslim Slim Bouguerra added a comment - https://reviews.apache.org/r/62262/
            githubbot ASF GitHub Bot added a comment -

            GitHub user b-slim opened a pull request:

            https://github.com/apache/hive/pull/249

            HIVE-17523 Fix insert into bug

            https://issues.apache.org/jira/browse/HIVE-17523

            You can merge this pull request into a Git repository by running:

            $ git pull https://github.com/b-slim/hive fix_insert_into

            Alternatively you can review and apply these changes as the patch at:

            https://github.com/apache/hive/pull/249.patch

            To close this pull request, make a commit to your master/trunk branch
            with (at least) the following in the commit message:

            This closes #249


            commit 579483ee878a16491d61c0eada5891880f507302
            Author: Slim Bouguerra <slim.bouguerra@gmail.com>
            Date: 2017-09-09T00:12:14Z

            set max tries to zero to make test faster.

            Change-Id: Ia1523f3b565f4b08c76067fa0bac32d5171fb46e

            commit ca5c7ccb1cc41dc3c8c76ccefb2d05b7d096a4ca
            Author: Slim Bouguerra <slim.bouguerra@gmail.com>
            Date: 2017-09-11T22:39:40Z

            Make insert into use data segment pusher to avoid duplication of logic some refactoring of exeception logging and handeling

            Change-Id: I7bd8f29a83720f4cfba338acf27fb85b9774eafe

            commit 3f2dd91c04839df7c068e1608b3e2af98babdc11
            Author: Slim Bouguerra <slim.bouguerra@gmail.com>
            Date: 2017-09-13T01:04:05Z

            cleaning and refactor

            Change-Id: I9e2b14e6e32af095d2c4d2f9f1fbe8a9cced30ad


            githubbot ASF GitHub Bot added a comment - GitHub user b-slim opened a pull request: https://github.com/apache/hive/pull/249 HIVE-17523 Fix insert into bug https://issues.apache.org/jira/browse/HIVE-17523 You can merge this pull request into a Git repository by running: $ git pull https://github.com/b-slim/hive fix_insert_into Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/249.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #249 commit 579483ee878a16491d61c0eada5891880f507302 Author: Slim Bouguerra <slim.bouguerra@gmail.com> Date: 2017-09-09T00:12:14Z set max tries to zero to make test faster. Change-Id: Ia1523f3b565f4b08c76067fa0bac32d5171fb46e commit ca5c7ccb1cc41dc3c8c76ccefb2d05b7d096a4ca Author: Slim Bouguerra <slim.bouguerra@gmail.com> Date: 2017-09-11T22:39:40Z Make insert into use data segment pusher to avoid duplication of logic some refactoring of exeception logging and handeling Change-Id: I7bd8f29a83720f4cfba338acf27fb85b9774eafe commit 3f2dd91c04839df7c068e1608b3e2af98babdc11 Author: Slim Bouguerra <slim.bouguerra@gmail.com> Date: 2017-09-13T01:04:05Z cleaning and refactor Change-Id: I9e2b14e6e32af095d2c4d2f9f1fbe8a9cced30ad

            Mostly looks good. Few minor comments on RB

            ashutoshc Ashutosh Chauhan added a comment - Mostly looks good. Few minor comments on RB
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12886768/HIVE-17523.patch

            SUCCESS: +1 due to 1 test(s) being added or modified.

            ERROR: -1 due to 8 failed/errored test(s), 11041 tests executed
            Failed tests:

            TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
            TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61)
            org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156)
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89)
            org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
            org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215)
            

            Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6804/testReport
            Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6804/console
            Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6804/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 8 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12886768 - PreCommit-HIVE-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12886768/HIVE-17523.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 8 failed/errored test(s), 11041 tests executed Failed tests: TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6804/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6804/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6804/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12886768 - PreCommit-HIVE-Build

            +1

            ashutoshc Ashutosh Chauhan added a comment - +1
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12889393/HIVE-17523.2.patch

            SUCCESS: +1 due to 1 test(s) being added or modified.

            ERROR: -1 due to 8 failed/errored test(s), 11089 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=232)
            org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=232)
            org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162)
            org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=157)
            org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=171)
            org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=101)
            org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=236)
            org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query64] (batchId=236)
            

            Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7022/testReport
            Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7022/console
            Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7022/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 8 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12889393 - PreCommit-HIVE-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12889393/HIVE-17523.2.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 8 failed/errored test(s), 11089 tests executed Failed tests: org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown] (batchId=232) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=232) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=157) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=171) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=101) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=236) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query64] (batchId=236) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7022/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7022/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7022/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12889393 - PreCommit-HIVE-Build

            Pushed to master. Thanks, Slim!

            ashutoshc Ashutosh Chauhan added a comment - Pushed to master. Thanks, Slim!

            This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.

            ashutoshc Ashutosh Chauhan added a comment - This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.
            githubbot ASF GitHub Bot added a comment -

            Github user b-slim closed the pull request at:

            https://github.com/apache/hive/pull/249

            githubbot ASF GitHub Bot added a comment - Github user b-slim closed the pull request at: https://github.com/apache/hive/pull/249

            People

              bslim Slim Bouguerra
              bslim Slim Bouguerra
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: