Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18164

Hive2 select with group by error if transactional = true

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: HiveServer2, Transactions
    • Labels:
      None
    • Environment:

      Hortonworks HDP-2.6.3.0

    • Target Version/s:

      Description

      Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235)
      Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235)
      0: jdbc:hive2://serv01:2181,ks-> select sum(destination),messagetype from t1.cdr where hday='2017-09-14' group by messagetype;
      INFO : Session is already open
      INFO : Dag name: select sum(destination),messag...messagetype(Stage-1)
      ERROR : Status: Failed
      ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1511771679762_0301_2_00, diagnostics=[Vertex vertex_1511771679762_0301_2_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: cdr initializer failed, vertex=vertex_1511771679762_0301_2_00 [Map 1], java.lang.RuntimeException: serious problem
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1277)
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1304)
      at org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat.getSplits(BucketizedHiveInputFormat.java:141)
      at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:448)
      at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:300)
      at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
      at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
      at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
      at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)
      at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with base_
      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1254)
      ... 16 more
      Caused by: java.lang.IllegalArgumentException: delta_16881612_29766798 does not start with base_
      at org.apache.hadoop.hive.ql.io.AcidUtils.parseBase(AcidUtils.java:190)
      at org.apache.hadoop.hive.ql.io.AcidUtils.parseBaseBucketFilename(AcidUtils.java:221)
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.callInternal(OrcInputFormat.java:804)
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.access$600(OrcInputFormat.java:747)
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:772)
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator$1.run(OrcInputFormat.java:769)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:769)
      at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:747)
      ... 4 more
      ]

      Error occur if delta_* present : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

      [serv01]$ hdfs dfs -ls /warehouse/t1/cdr/hday=2017-09-14
      Found 15 items
      drwxrwxrwx - hive hdfs 0 2017-09-16 11:29 /warehouse/t1/cdr/hday=2017-09-14/base_16881497
      drwxrwxrwx - hive hdfs 0 2017-10-21 18:42 /warehouse/t1/cdr/hday=2017-09-14/delta_16881612_29766798
      drwxr-xr-x - hive hdfs 0 2017-10-22 17:48 /warehouse/t1/cdr/hday=2017-09-14/delta_30628231_30628231_0000
      drwxr-xr-x - hive hdfs 0 2017-10-26 18:06 /warehouse/t1/cdr/hday=2017-09-14/delta_33418590_33418590_0000
      drwxr-xr-x - hive hdfs 0 2017-10-27 16:23 /warehouse/t1/cdr/hday=2017-09-14/delta_33540229_33540229_0000
      drwxr-xr-x - hive hdfs 0 2017-10-27 16:33 /warehouse/t1/cdr/hday=2017-09-14/delta_33541305_33541305_0000
      drwxr-xr-x - hive hdfs 0 2017-10-31 12:40 /warehouse/t1/cdr/hday=2017-09-14/delta_34016509_34016509_0000
      drwxr-xr-x - hive hdfs 0 2017-10-31 13:30 /warehouse/t1/cdr/hday=2017-09-14/delta_34025608_34025608_0000
      drwxr-xr-x - hive hdfs 0 2017-10-31 14:19 /warehouse/t1/cdr/hday=2017-09-14/delta_34033668_34033668_0000
      drwxr-xr-x - hive hdfs 0 2017-11-01 21:38 /warehouse/t1/cdr/hday=2017-09-14/delta_34219785_34219785_0000
      drwxr-xr-x - hive hdfs 0 2017-11-02 11:20 /warehouse/t1/cdr/hday=2017-09-14/delta_34292833_34292833_0000
      drwxr-xr-x - hive hdfs 0 2017-11-10 09:52 /warehouse/t1/cdr/hday=2017-09-14/delta_35449030_35449030_0000
      drwxr-xr-x - hive hdfs 0 2017-11-10 13:07 /warehouse/t1/cdr/hday=2017-09-14/delta_35472185_35472185_0000
      drwxr-xr-x - hive hdfs 0 2017-11-13 19:07 /warehouse/t1/cdr/hday=2017-09-14/delta_35944544_35944544_0000
      drwxr-xr-x - hive hdfs 0 2017-11-21 12:37 /warehouse/t1/cdr/hday=2017-09-14/delta_36820930_36820930_0000

      Workaround:

      ALTER TABLE .. SET TBLPROPERTIES ('compactorthreshold.hive.compactor.delta.num.threshold'='1') - and wait done compactors working

      OR

      set hive.execution.engine=mr;

      Question :

      exist any other workaround for run select with TEZ ?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Dmitro-Vasilenko Dmitro-Vasilenko
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Due:
                Created:
                Updated: