[KYLIN-1745] Java heap space errors for all builds in Kylin 1.5.2

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v1.5.2
• Fix Version/s: v1.5.2.1
• Assignee: Shao Feng Shi
• Reporter: Nick Muerdter
• Created: 29/May/16 02:08
• Updated: 04/Feb/18 11:35
• Resolved: 13/Jun/16 06:38

Description

When trying to build any cubes in Kylin 1.5.2, I get Java heap space errors on step 2 (Extract Fact Table Distinct Columns). I experience the same heap space error on my own cube, as well as using the sample cube data (from sample.sh).

I'm testing on a fairly stock HDP-2.2.9.0-3393 installation. I've tried tweaking the memory settings, but even with the map memory tuned to allow 8GB, I still get the same heap space error even for the small sample data (where 8GB seems like it should be more than plenty).

If I try the same build against the sample data on a fresh 1.5.1 installation, the cube builds as expected, but with a fresh 1.5.2 installation, I get these heap space errors on any build attempts.

Here's the backtrace from the failed map job:

2016-05-29 01:51:57,910 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2016-05-29 01:51:57,977 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2016-05-29 01:51:57,977 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2016-05-29 01:51:57,984 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2016-05-29 01:51:58,017 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1464481246919_0079, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@548e6d58)
2016-05-29 01:51:58,045 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2016-05-29 01:51:58,249 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /hadoop/yarn/local/usercache/api-umbrella/appcache/application_1464481246919_0079
2016-05-29 01:51:58,443 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2016-05-29 01:51:58,792 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2016-05-29 01:51:58,801 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2016-05-29 01:51:59,078 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: org.apache.hive.hcatalog.mapreduce.HCatSplit@12dae582
2016-05-29 01:51:59,218 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:983)
	at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)
	at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Any ideas? Let me know if I can supply any other details.

Thanks!