Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-2502

"Create flat table" and "redistribute table" steps don't show YARN application link

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.6.0
    • Fix Version/s: v2.0.0
    • Component/s: Job Engine
    • Labels:
      None

      Description

      From some version, Kylin doesn't show the tracking url for the hive steps. Need take a look.

      A sample output is:

      Create and distribute table, cmd: 
      hive -e "SET dfs.replication=2;
      SET hive.exec.compress.output=true;
      SET hive.auto.convert.join=true;
      SET hive.auto.convert.join.noconditionaltask=true;
      SET hive.auto.convert.join.noconditionaltask.size=100000000;
      SET mapreduce.job.split.metainfo.maxsize=-1;
      SET hive.stats.autogather=true;
      SET hive.merge.mapfiles=false;
      SET hive.merge.mapredfiles=false;
      
      USE default;
      DROP TABLE IF EXISTS kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e;
      CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e
      (
      KYLIN_SALES_TRANS_ID bigint
      ,KYLIN_SALES_PART_DT date
      ,KYLIN_SALES_LEAF_CATEG_ID bigint
      ,KYLIN_SALES_LSTG_SITE_ID int
      ,KYLIN_CATEGORY_GROUPINGS_META_CATEG_NAME string
      ,KYLIN_CATEGORY_GROUPINGS_CATEG_LVL2_NAME string
      ,KYLIN_CATEGORY_GROUPINGS_CATEG_LVL3_NAME string
      ,KYLIN_SALES_LSTG_FORMAT_NAME string
      ,KYLIN_SALES_SELLER_ID bigint
      ,KYLIN_SALES_BUYER_ID bigint
      ,BUYER_ACCOUNT_ACCOUNT_BUYER_LEVEL int
      ,SELLER_ACCOUNT_ACCOUNT_SELLER_LEVEL int
      ,BUYER_ACCOUNT_ACCOUNT_COUNTRY string
      ,SELLER_ACCOUNT_ACCOUNT_COUNTRY string
      ,BUYER_COUNTRY_NAME string
      ,SELLER_COUNTRY_NAME string
      ,KYLIN_SALES_OPS_USER_ID string
      ,KYLIN_SALES_OPS_REGION string
      ,KYLIN_SALES_PRICE decimal(19,4)
      )
      STORED AS SEQUENCEFILE
      LOCATION 'hdfs:///kylin/kylin_metadata/kylin-2d5c1178-c6f6-4b50-8937-8e5e3b39227e/kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e';
      SET dfs.replication=2;
      SET hive.exec.compress.output=true;
      SET hive.auto.convert.join=true;
      SET hive.auto.convert.join.noconditionaltask=true;
      SET hive.auto.convert.join.noconditionaltask.size=100000000;
      SET mapreduce.job.split.metainfo.maxsize=-1;
      SET hive.stats.autogather=true;
      SET hive.merge.mapfiles=false;
      SET hive.merge.mapredfiles=false;
      INSERT OVERWRITE TABLE kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e SELECT
      KYLIN_SALES.TRANS_ID
      ,KYLIN_SALES.PART_DT
      ,KYLIN_SALES.LEAF_CATEG_ID
      ,KYLIN_SALES.LSTG_SITE_ID
      ,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME
      ,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
      ,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME
      ,KYLIN_SALES.LSTG_FORMAT_NAME
      ,KYLIN_SALES.SELLER_ID
      ,KYLIN_SALES.BUYER_ID
      ,BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL
      ,SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL
      ,BUYER_ACCOUNT.ACCOUNT_COUNTRY
      ,SELLER_ACCOUNT.ACCOUNT_COUNTRY
      ,BUYER_COUNTRY.NAME
      ,SELLER_COUNTRY.NAME
      ,KYLIN_SALES.OPS_USER_ID
      ,KYLIN_SALES.OPS_REGION
      ,KYLIN_SALES.PRICE
      FROM DEFAULT.KYLIN_SALES as KYLIN_SALES 
      INNER JOIN DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
      ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
      INNER JOIN DEFAULT.KYLIN_CATEGORY_GROUPINGS as KYLIN_CATEGORY_GROUPINGS
      ON KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
      INNER JOIN DEFAULT.KYLIN_ACCOUNT as BUYER_ACCOUNT
      ON KYLIN_SALES.BUYER_ID = BUYER_ACCOUNT.ACCOUNT_ID
      INNER JOIN DEFAULT.KYLIN_ACCOUNT as SELLER_ACCOUNT
      ON KYLIN_SALES.SELLER_ID = SELLER_ACCOUNT.ACCOUNT_ID
      INNER JOIN DEFAULT.KYLIN_COUNTRY as BUYER_COUNTRY
      ON BUYER_ACCOUNT.ACCOUNT_COUNTRY = BUYER_COUNTRY.COUNTRY
      INNER JOIN DEFAULT.KYLIN_COUNTRY as SELLER_COUNTRY
      ON SELLER_ACCOUNT.ACCOUNT_COUNTRY = SELLER_COUNTRY.COUNTRY
      WHERE (KYLIN_SALES.PART_DT >= '2012-01-01' AND KYLIN_SALES.PART_DT < '2017-03-01')
      ;
      
      "
      WARNING: Use "yarn jar" to launch YARN applications.
      
      Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties
      OK
      Time taken: 1.941 seconds
      OK
      Time taken: 5.394 seconds
      OK
      Time taken: 4.008 seconds
      Query ID = root_20170306144241_724ee8c1-fb9e-4cb5-b37e-04a87fc114e5
      Total jobs = 1
      Launching Job 1 out of 1
      
      
      Status: Running (Executing on YARN cluster with App id application_1488805575687_0013)
      
      Map 1: -/-	Map 2: -/-	Map 3: -/-	Map 4: -/-	Map 5: -/-	Map 6: -/-	Map 7: -/-	
      Map 1: 0/1	Map 2: 0/1	Map 3: 0/1	Map 4: 0/1	Map 5: 0/1	Map 6: 0/1	Map 7: 0/1	
      Map 1: 0/1	Map 2: 0/1	Map 3: 0/1	Map 4: 0/1	Map 5: 0/1	Map 6: 0/1	Map 7: 0/1	
      Map 1: 0/1	Map 2: 0/1	Map 3: 0(+1)/1	Map 4: 0/1	Map 5: 0(+1)/1	Map 6: 0/1	Map 7: 0/1	
      Map 1: 0/1	Map 2: 0/1	Map 3: 0(+1)/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 0(+1)/1	Map 7: 0/1	
      Map 1: 0/1	Map 2: 0/1	Map 3: 0(+1)/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 0(+1)/1	Map 7: 0/1	
      Map 1: 0/1	Map 2: 0/1	Map 3: 1/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 0(+1)/1	Map 7: 0/1	
      Map 1: 0(+1)/1	Map 2: 0/1	Map 3: 1/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 1/1	Map 7: 0(+1)/1	
      Map 1: 0(+1)/1	Map 2: 0(+1)/1	Map 3: 1/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 1/1	Map 7: 1/1	
      Map 1: 0(+1)/1	Map 2: 0(+1)/1	Map 3: 1/1	Map 4: 1/1	Map 5: 1/1	Map 6: 1/1	Map 7: 1/1	
      Map 1: 1/1	Map 2: 0(+1)/1	Map 3: 1/1	Map 4: 1/1	Map 5: 1/1	Map 6: 1/1	Map 7: 1/1	
      Map 1: 1/1	Map 2: 1/1	Map 3: 1/1	Map 4: 1/1	Map 5: 1/1	Map 6: 1/1	Map 7: 1/1	
      Loading data to table default.kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e
      Table default.kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e stats: [numFiles=1, numRows=10000, totalSize=302827, rawDataSize=1519831]
      OK
      Time taken: 26.551 seconds
      
      

        Attachments

          Activity

            People

            • Assignee:
              shaofengshi Shao Feng Shi
              Reporter:
              shaofengshi Shao Feng Shi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: