[KYLIN-2502] "Create flat table" and "redistribute table" steps don't show YARN application link - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: v1.6.0
Fix Version/s: v2.0.0
Component/s: Job Engine
Labels:
None

Description

From some version, Kylin doesn't show the tracking url for the hive steps. Need take a look.

A sample output is:

Create and distribute table, cmd: 
hive -e "SET dfs.replication=2;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=100000000;
SET mapreduce.job.split.metainfo.maxsize=-1;
SET hive.stats.autogather=true;
SET hive.merge.mapfiles=false;
SET hive.merge.mapredfiles=false;

USE default;
DROP TABLE IF EXISTS kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e;
CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e
(
KYLIN_SALES_TRANS_ID bigint
,KYLIN_SALES_PART_DT date
,KYLIN_SALES_LEAF_CATEG_ID bigint
,KYLIN_SALES_LSTG_SITE_ID int
,KYLIN_CATEGORY_GROUPINGS_META_CATEG_NAME string
,KYLIN_CATEGORY_GROUPINGS_CATEG_LVL2_NAME string
,KYLIN_CATEGORY_GROUPINGS_CATEG_LVL3_NAME string
,KYLIN_SALES_LSTG_FORMAT_NAME string
,KYLIN_SALES_SELLER_ID bigint
,KYLIN_SALES_BUYER_ID bigint
,BUYER_ACCOUNT_ACCOUNT_BUYER_LEVEL int
,SELLER_ACCOUNT_ACCOUNT_SELLER_LEVEL int
,BUYER_ACCOUNT_ACCOUNT_COUNTRY string
,SELLER_ACCOUNT_ACCOUNT_COUNTRY string
,BUYER_COUNTRY_NAME string
,SELLER_COUNTRY_NAME string
,KYLIN_SALES_OPS_USER_ID string
,KYLIN_SALES_OPS_REGION string
,KYLIN_SALES_PRICE decimal(19,4)
)
STORED AS SEQUENCEFILE
LOCATION 'hdfs:///kylin/kylin_metadata/kylin-2d5c1178-c6f6-4b50-8937-8e5e3b39227e/kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e';
SET dfs.replication=2;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=100000000;
SET mapreduce.job.split.metainfo.maxsize=-1;
SET hive.stats.autogather=true;
SET hive.merge.mapfiles=false;
SET hive.merge.mapredfiles=false;
INSERT OVERWRITE TABLE kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e SELECT
KYLIN_SALES.TRANS_ID
,KYLIN_SALES.PART_DT
,KYLIN_SALES.LEAF_CATEG_ID
,KYLIN_SALES.LSTG_SITE_ID
,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME
,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME
,KYLIN_SALES.LSTG_FORMAT_NAME
,KYLIN_SALES.SELLER_ID
,KYLIN_SALES.BUYER_ID
,BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL
,SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL
,BUYER_ACCOUNT.ACCOUNT_COUNTRY
,SELLER_ACCOUNT.ACCOUNT_COUNTRY
,BUYER_COUNTRY.NAME
,SELLER_COUNTRY.NAME
,KYLIN_SALES.OPS_USER_ID
,KYLIN_SALES.OPS_REGION
,KYLIN_SALES.PRICE
FROM DEFAULT.KYLIN_SALES as KYLIN_SALES 
INNER JOIN DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
INNER JOIN DEFAULT.KYLIN_CATEGORY_GROUPINGS as KYLIN_CATEGORY_GROUPINGS
ON KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
INNER JOIN DEFAULT.KYLIN_ACCOUNT as BUYER_ACCOUNT
ON KYLIN_SALES.BUYER_ID = BUYER_ACCOUNT.ACCOUNT_ID
INNER JOIN DEFAULT.KYLIN_ACCOUNT as SELLER_ACCOUNT
ON KYLIN_SALES.SELLER_ID = SELLER_ACCOUNT.ACCOUNT_ID
INNER JOIN DEFAULT.KYLIN_COUNTRY as BUYER_COUNTRY
ON BUYER_ACCOUNT.ACCOUNT_COUNTRY = BUYER_COUNTRY.COUNTRY
INNER JOIN DEFAULT.KYLIN_COUNTRY as SELLER_COUNTRY
ON SELLER_ACCOUNT.ACCOUNT_COUNTRY = SELLER_COUNTRY.COUNTRY
WHERE (KYLIN_SALES.PART_DT >= '2012-01-01' AND KYLIN_SALES.PART_DT < '2017-03-01')
;

"
WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties
OK
Time taken: 1.941 seconds
OK
Time taken: 5.394 seconds
OK
Time taken: 4.008 seconds
Query ID = root_20170306144241_724ee8c1-fb9e-4cb5-b37e-04a87fc114e5
Total jobs = 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id application_1488805575687_0013)

Map 1: -/-	Map 2: -/-	Map 3: -/-	Map 4: -/-	Map 5: -/-	Map 6: -/-	Map 7: -/-	
Map 1: 0/1	Map 2: 0/1	Map 3: 0/1	Map 4: 0/1	Map 5: 0/1	Map 6: 0/1	Map 7: 0/1	
Map 1: 0/1	Map 2: 0/1	Map 3: 0/1	Map 4: 0/1	Map 5: 0/1	Map 6: 0/1	Map 7: 0/1	
Map 1: 0/1	Map 2: 0/1	Map 3: 0(+1)/1	Map 4: 0/1	Map 5: 0(+1)/1	Map 6: 0/1	Map 7: 0/1	
Map 1: 0/1	Map 2: 0/1	Map 3: 0(+1)/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 0(+1)/1	Map 7: 0/1	
Map 1: 0/1	Map 2: 0/1	Map 3: 0(+1)/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 0(+1)/1	Map 7: 0/1	
Map 1: 0/1	Map 2: 0/1	Map 3: 1/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 0(+1)/1	Map 7: 0/1	
Map 1: 0(+1)/1	Map 2: 0/1	Map 3: 1/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 1/1	Map 7: 0(+1)/1	
Map 1: 0(+1)/1	Map 2: 0(+1)/1	Map 3: 1/1	Map 4: 0(+1)/1	Map 5: 0(+1)/1	Map 6: 1/1	Map 7: 1/1	
Map 1: 0(+1)/1	Map 2: 0(+1)/1	Map 3: 1/1	Map 4: 1/1	Map 5: 1/1	Map 6: 1/1	Map 7: 1/1	
Map 1: 1/1	Map 2: 0(+1)/1	Map 3: 1/1	Map 4: 1/1	Map 5: 1/1	Map 6: 1/1	Map 7: 1/1	
Map 1: 1/1	Map 2: 1/1	Map 3: 1/1	Map 4: 1/1	Map 5: 1/1	Map 6: 1/1	Map 7: 1/1	
Loading data to table default.kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e
Table default.kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e stats: [numFiles=1, numRows=10000, totalSize=302827, rawDataSize=1519831]
OK
Time taken: 26.551 seconds

"Create flat table" and "redistribute table" steps don't show YARN application link

Details

Description

Attachments

Activity

People

Dates