Log Type: prelaunch.err Log Upload Time: Wed Dec 19 18:09:15 +0800 2018 Log Length: 0 Log Type: prelaunch.out Log Upload Time: Wed Dec 19 18:09:15 +0800 2018 Log Length: 70 Setting up env variables Setting up job resources Launching container Log Type: stderr Log Upload Time: Wed Dec 19 18:09:15 +0800 2018 Log Length: 0 Log Type: stdout Log Upload Time: Wed Dec 19 18:09:15 +0800 2018 Log Length: 0 Log Type: syslog Log Upload Time: Wed Dec 19 18:09:15 +0800 2018 Log Length: 95728 Showing 4096 bytes of 95728 total. Click here for the full log. Reducer: Received value: [10001, 9846, 8336, 8330, 8327] 2018-12-19 18:09:07,953 ERROR [main] org.apache.kylin.engine.mr.KylinReducer: java.lang.IllegalArgumentException: Unsupported data type array at org.apache.kylin.metadata.datatype.DataTypeOrder.getInstance(DataTypeOrder.java:56) at org.apache.kylin.metadata.datatype.DataType.getOrder(DataType.java:230) at org.apache.kylin.metadata.datatype.DataType.compare(DataType.java:236) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doReduce(FactDistinctColumnsReducer.java:162) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doReduce(FactDistinctColumnsReducer.java:59) at org.apache.kylin.engine.mr.KylinReducer.reduce(KylinReducer.java:77) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2018-12-19 18:09:07,955 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do cleanup, available memory: 493m 2018-12-19 18:09:07,963 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2 2018-12-19 18:09:07,963 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2018-12-19 18:09:08,068 INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer: write dimension range info for col : IMPRESSION_CLICK_ID minValue:[10001, 9066] maxValue:[10001, 9066] 2018-12-19 18:09:08,073 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2 2018-12-19 18:09:08,074 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2018-12-19 18:09:08,229 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Total rows: 2 2018-12-19 18:09:08,229 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException: Unsupported data type array at org.apache.kylin.metadata.datatype.DataTypeOrder.getInstance(DataTypeOrder.java:56) at org.apache.kylin.metadata.datatype.DataType.getOrder(DataType.java:230) at org.apache.kylin.metadata.datatype.DataType.compare(DataType.java:236) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doReduce(FactDistinctColumnsReducer.java:162) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doReduce(FactDistinctColumnsReducer.java:59) at org.apache.kylin.engine.mr.KylinReducer.reduce(KylinReducer.java:77) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2018-12-19 18:09:08,231 INFO [main] org.apache.hadoop.mapred.Task: Running cleanup for the task 2018-12-19 18:09:08,338 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ReduceTask metrics system... 2018-12-19 18:09:08,339 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system stopped. 2018-12-19 18:09:08,339 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system shutdown complete. Log Type: syslog.shuffle Log Upload Time: Wed Dec 19 18:09:15 +0800 2018 Log Length: 3312 2018-12-19 18:09:07,017 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: MergerManager: memoryLimit=535088320, maxSingleShuffleLimit=133772080, mergeThreshold=353158304, ioSortFactor=64, memToMemMergeOutputsThreshold=64 2018-12-19 18:09:07,019 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1544061294238_0030_r_000002_3 Thread started: EventFetcher for fetching Map Completion Events 2018-12-19 18:09:07,026 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1544061294238_0030_r_000002_3: Got 2 new map-outputs 2018-12-19 18:09:07,190 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1544061294238_0030_m_000001_0 decomp: 1344148 len: 335958 to MEMORY 2018-12-19 18:09:07,194 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1344148 bytes from map-output for attempt_1544061294238_0030_m_000001_0 2018-12-19 18:09:07,194 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1344148, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->1344148 2018-12-19 18:09:07,195 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1544061294238_0030_m_000000_0 decomp: 1338129 len: 334094 to MEMORY 2018-12-19 18:09:07,197 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 1338129 bytes from map-output for attempt_1544061294238_0030_m_000000_0 2018-12-19 18:09:07,197 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1338129, inMemoryMapOutputs.size() -> 2, commitMemory -> 1344148, usedMemory ->2682277 2018-12-19 18:09:07,198 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: pr-bigdata-cdh02:13562 freed by fetcher#1 in 168ms 2018-12-19 18:09:07,198 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning 2018-12-19 18:09:07,202 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs 2018-12-19 18:09:07,209 INFO [main] org.apache.hadoop.mapred.Merger: Merging 2 sorted segments 2018-12-19 18:09:07,210 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2682223 bytes 2018-12-19 18:09:07,271 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 2 segments, 2682277 bytes to disk to satisfy reduce memory limit 2018-12-19 18:09:07,272 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, 526273 bytes from disk 2018-12-19 18:09:07,273 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 2018-12-19 18:09:07,273 INFO [main] org.apache.hadoop.mapred.Merger: Merging 1 sorted segments 2018-12-19 18:09:07,278 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2682257 bytes