Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1537

Failed to create flat hive table, when name is too long

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: v1.5.0
    • Fix Version/s: v1.5.1
    • Component/s: Job Engine
    • Labels:
      None

      Description

      Run Kylin regression test on HDP 2.4, Hive version 1.2.1.2.4; A cube couldn't build, but others can; Got error in the first step:

      OS command error exit with 12 -- hive -e "USE default;
      DROP TABLE IF EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20150101000000;
      
      CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20150101000000
      (
      DEFAULT_TEST_KYLIN_FACT_CAL_DT date
      ,DEFAULT_TEST_KYLIN_FACT_LEAF_CATEG_ID bigint
      ,DEFAULT_TEST_KYLIN_FACT_LSTG_SITE_ID int
      ,DEFAULT_TEST_CATEGORY_GROUPINGS_META_CATEG_NAME string
      ,DEFAULT_TEST_CATEGORY_GROUPINGS_CATEG_LVL2_NAME string
      ,DEFAULT_TEST_CATEGORY_GROUPINGS_CATEG_LVL3_NAME string
      ,DEFAULT_TEST_KYLIN_FACT_LSTG_FORMAT_NAME string
      ,DEFAULT_TEST_KYLIN_FACT_SLR_SEGMENT_CD smallint
      ,DEFAULT_TEST_KYLIN_FACT_SELLER_ID bigint
      ,DEFAULT_TEST_KYLIN_FACT_PRICE decimal(19,4)
      ,DEFAULT_TEST_KYLIN_FACT_ITEM_COUNT int
      )
      ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
      STORED AS SEQUENCEFILE
      LOCATION '/kylin/kylin_metadata/kylin-a28a48e4-83a2-4b71-b06d-9a281540507d/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20150101000000';
      
      SET dfs.replication=2;
      SET hive.exec.compress.output=true;
      SET hive.auto.convert.join.noconditionaltask=true;
      SET hive.auto.convert.join.noconditionaltask.size=300000000;
      SET hive.merge.mapfiles=true;
      SET hive.merge.mapredfiles=true;
      SET hive.merge.size.per.task=64000000;
      INSERT OVERWRITE TABLE kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20150101000000 SELECT
      TEST_KYLIN_FACT.CAL_DT
      ,TEST_KYLIN_FACT.LEAF_CATEG_ID
      ,TEST_KYLIN_FACT.LSTG_SITE_ID
      ,TEST_CATEGORY_GROUPINGS.META_CATEG_NAME
      ,TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
      ,TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME
      ,TEST_KYLIN_FACT.LSTG_FORMAT_NAME
      ,TEST_KYLIN_FACT.SLR_SEGMENT_CD
      ,TEST_KYLIN_FACT.SELLER_ID
      ,TEST_KYLIN_FACT.PRICE
      ,TEST_KYLIN_FACT.ITEM_COUNT
      FROM DEFAULT.TEST_KYLIN_FACT as TEST_KYLIN_FACT 
      INNER JOIN EDW.TEST_CAL_DT as TEST_CAL_DT
      ON TEST_KYLIN_FACT.CAL_DT = TEST_CAL_DT.CAL_DT
      INNER JOIN DEFAULT.TEST_CATEGORY_GROUPINGS as TEST_CATEGORY_GROUPINGS
      ON TEST_KYLIN_FACT.LEAF_CATEG_ID = TEST_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_CATEGORY_GROUPINGS.SITE_ID
      INNER JOIN EDW.TEST_SITES as TEST_SITES
      ON TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_SITES.SITE_ID
      INNER JOIN EDW.TEST_SELLER_TYPE_DIM as TEST_SELLER_TYPE_DIM
      ON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD
      WHERE (TEST_KYLIN_FACT.CAL_DT < '2015-01-01')
      ;
      
      "
      WARNING: Use "yarn jar" to launch YARN applications.
      
      Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties
      OK
      Time taken: 1.812 seconds
      OK
      Time taken: 5.622 seconds
      FAILED: Hive Internal Error: org.codehaus.jettison.json.JSONException(JSONArray[0] is not a JSONObject.)
      org.codehaus.jettison.json.JSONException: JSONArray[0] is not a JSONObject.
      	at org.codehaus.jettison.json.JSONArray.getJSONObject(JSONArray.java:258)
      	at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.getEntityReferenceFromDSL(HiveMetaStoreBridge.java:193)
      	at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.getTableReference(HiveMetaStoreBridge.java:221)
      	at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerTable(HiveMetaStoreBridge.java:276)
      	at org.apache.atlas.hive.hook.HiveHook.handleCreateTable(HiveHook.java:271)
      	at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:205)
      	at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:172)
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1585)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:314)
      	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:711)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      

      Reason: the table name is too long; Kylin should use a name which won't exceed Hive's limit.

        Attachments

          Activity

            People

            • Assignee:
              Shaofengshi Shaofeng SHI
              Reporter:
              Shaofengshi Shaofeng SHI
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: