Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3695

Error while creating hive table through Kylin build cube with mysql imported tables

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: v2.5.0
    • Fix Version/s: v2.6.0
    • Component/s: Integration
    • Labels:
      None

      Description

      HI I am trying to build a cube with Kylin, the data gets souced fine from sqoop but the next step for creating hive tables fails . Looking at the command being fired it looks weird as the create statement looks good to me .

      I think the issue is with DOUBLE types as when I remove the same the create statement works fine . Can someone please help .

      I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0

      The errors logs with commands as as below, the table is a msql table which had columns with DOUBLE type

      Command

      hive -e "USE default; DROP TABLE IF EXISTS kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 ( HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22) ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE LOCATION 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368'; ALTER TABLE kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true

      Error is as below

      OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds MismatchedTokenException(334!=347) at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617) at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179) at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) FAILED: ParseException line 15:42 mismatched input '(' expecting ) near 'double' in create table statement

      hadoop hive kylin
       

      More details here  https://stackoverflow.com/questions/53377623/error-while-creating-hive-table-through-kylin-build-cube . Though the same got solved when I used DECIMAL type with precision .

       

      Also observer that if you use DECIMAL(10) it does not work unless it has a precision while building mode for the same it gives error as tries to use precision as undefined ie . DECIMAL(10,undefined)

        Attachments

        1. image-2018-12-04-15-53-49-983.png
          131 kB
          Xiaoxiang Yu

          Issue Links

            Activity

              People

              • Assignee:
                hit_lacus Xiaoxiang Yu
                Reporter:
                gauravbrills Gaurav Rawat
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: