[KYLIN-2116] when hive field delimitor exists in table field values, fields order is wrong

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: v1.5.2
• Fix Version/s: v1.6.0
• Component/s: Job Engine
• Assignee: Dong Li
• Reporter: yubo
• Created: 21/Oct/16 02:12
• Updated: 26/Nov/16 08:12
• Resolved: 31/Oct/16 05:42

Description

in #1 step when creating temp hive table,there is delimitor

ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'

when this delimitor exists in some fields, fields order is wrong

test details:

when we search with same sql, different results are returned as below.
25 in kylin and 24 in hive.

We guess there maybe some problem within #2 Step(Extract Fact Table Distinct Columns) when building cube.

1 search in kylin
select distinct visit_hour from KYLIN_REPORT_DB.session_behavior_channel_oms where visit_date >= '2016-10-19' and visit_date <= '2016-10-19'
Results (25)
19
17
18
15
16
13
14
11
12
21
神马搜索
20
08
09
04
22
05
23
06
07
00
01
02
03
10

2. #2 Step Name: Extract Fact Table Distinct Columns output

hadoop fs -cat /kylin/kylin_metadata/kylin-e8bb517d-6c29-4f89-a83e-871e142e3d48/channel_first_stage_flow_cube/fact_distinct_columns/VISIT_HOUR
00
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
神马搜索

3. hive table

hive -e " select distinct visit_hour from KYLIN_REPORT_DB.session_behavior_channel_oms where visit_date >= '2016-10-19' and visit_date <= '2016-10-19' "
WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in file:/etc/hive/2.3.4.0-3485/0/hive-log4j.properties
Query ID = hdfs_20161020164441_dcea3e55-1a8b-4f3a-9378-7dcda008001b
Total jobs = 1
Launching Job 1 out of 1

Status: Running (Executing on YARN cluster with App id application_1476342479107_13013)

--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 564 564 0 0 0 0
Reducer 2 ...... SUCCEEDED 15 15 0 0 0 0
--------------------------------------------------------------------------------
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 168.70 s
--------------------------------------------------------------------------------
OK
03
12
13
22
05
14
08
17
00
02
18
06
23
01
19
07
10
15
20
16
11
04
09
21
Time taken: 172.907 seconds, Fetched: 24 row(s)