Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20085

Druid-Hive (managed) table creation fails with strict managed table checks: Table is marked as a managed table but is not transactional

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 4.0.0-alpha-1
    • Hive, StorageHandler
    • None

    Description

      Druid-Hive (managed) table creation fails with strict managed table checks: Table is marked as a managed table but is not transactional

      drop table if exists calcs;
      create table calcs
      STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
      TBLPROPERTIES (
      "druid.segment.granularity" = "MONTH",
      "druid.query.granularity" = "DAY")
      AS SELECT
      cast(datetime0 as timestamp with local time zone) `__time`,
      key,
      str0, str1, str2, str3,
      date0, date1, date2, date3,
      time0, time1,
      datetime0, datetime1,
      zzz,
      cast(bool0 as string) bool0,
      cast(bool1 as string) bool1,
      cast(bool2 as string) bool2,
      cast(bool3 as string) bool3,
      int0, int1, int2, int3,
      num0, num1, num2, num3, num4
      from tableau_orc.calcs;
      
      2018-07-03 04:57:31,911|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : Status: Running (Executing on YARN cluster with App id application_1530592209763_0009)
      ...
      ...
      2018-07-03 04:57:36,334|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : SHUFFLE_BYTES_TO_MEM: 0
      2018-07-03 04:57:36,334|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : SHUFFLE_PHASE_TIME: 330
      2018-07-03 04:57:36,334|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : SPILLED_RECORDS: 17
      2018-07-03 04:57:36,334|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : TaskCounter_Reducer_2_OUTPUT_out_Reducer_2:
      2018-07-03 04:57:36,335|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : OUTPUT_RECORDS: 0
      2018-07-03 04:57:36,335|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : org.apache.hadoop.hive.llap.counters.LlapWmCounters:
      2018-07-03 04:57:36,335|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : GUARANTEED_QUEUED_NS: 0
      2018-07-03 04:57:36,335|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : GUARANTEED_RUNNING_NS: 0
      2018-07-03 04:57:36,335|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : SPECULATIVE_QUEUED_NS: 2162643606
      2018-07-03 04:57:36,335|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : SPECULATIVE_RUNNING_NS: 12151664909
      2018-07-03 04:57:36,335|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : Starting task [Stage-2:DEPENDENCY_COLLECTION] in serial mode
      2018-07-03 04:57:36,335|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : Starting task [Stage-0:MOVE] in serial mode
      2018-07-03 04:57:36,336|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : Moving data to directory hdfs://mycluster/warehouse/tablespace/managed/hive/druid_tableau.db/calcs from hdfs://mycluster/warehouse/tablespace/managed/hive/druid_tableau.db/.hive-staging_hive_2018-07-03_04-57-27_351_7124633902209008283-3/-ext-10002
      2018-07-03 04:57:36,336|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : Starting task [Stage-4:DDL] in serial mode
      2018-07-03 04:57:36,336|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Table druid_tableau.calcs failed strict managed table checks due to the following reason: Table is marked as a managed table but is not transactional.)
      2018-07-03 04:57:36,336|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|INFO : Completed executing command(queryId=hive_20180703045727_c39c40d2-7d4a-46c7-a36d-7925e7c4a788); Time taken: 6.794 seconds
      2018-07-03 04:57:36,337|INFO|Thread-721|machine.py:111 - tee_pipe()||aa121a45-29eb-48a8-8628-ae5368aa172d|Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Table druid_tableau.calcs failed strict managed table checks due to the following reason: Table is marked as a managed table but is not transactional.) (state=08S01,code=1)
      

      This will not allow druid tables to be managed.

      So its not direct to create Druid tables.

      while trying to modify things to external tables..we see below issues
      1) INSERT/ INSERT OVERWRITE/ DROP are supported by Hive managed tables (not external) , we have few tests which covers this.. what would be the course of action.

      2) Druid Hive kafka ingestion (where we will not have any datasource, we just get from Kafka topic) , here when declared as external tables we get error

      message:Datasource name should be specified using [druid.datasource] for external tables using Druid
      

      Attachments

        1. HIVE-20085.1.patch
          48 kB
          Nishant Bangarwa
        2. HIVE-20085.2.patch
          49 kB
          Nishant Bangarwa
        3. HIVE-20085.3.patch
          51 kB
          Nishant Bangarwa
        4. HIVE-20085.4.patch
          52 kB
          Nishant Bangarwa
        5. HIVE-20085.5.patch
          53 kB
          Nishant Bangarwa
        6. HIVE-20085.patch
          48 kB
          Nishant Bangarwa

        Issue Links

          Activity

            People

              nishantbangarwa Nishant Bangarwa
              dileep529 Dileep Kumar Chiguruvada
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: