[KYLIN-3963] Provide a custom serializer for PercentileCounter

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: v3.1.0
• Component/s: Flink Engine
• Assignee: vinoyang
• Reporter: vinoyang
• Created: 20/Apr/19 02:38
• Updated: 26/Jul/20 02:36
• Resolved: 04/Sep/19 06:54

Description

PercentileCounter contains an instance field of Type TDigest which is not serializable, so it is marked as transient. By default, it triggers a NPE when use Flink cube engine:

Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 8077ff0e0ff01a26f62b1babe90a34b7) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:475) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) at org.apache.kylin.engine.flink.FlinkCubingByLayer.execute(FlinkCubingByLayer.java:196) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 18 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265) ... 23 more 
Caused by: java.lang.NullPointerException at org.apache.kylin.measure.percentile.PercentileCounter.writeRegisters(PercentileCounter.java:68) at org.apache.kylin.measure.percentile.PercentileSerializer.serialize(PercentileSerializer.java:70) at org.apache.kylin.measure.percentile.PercentileSerializer.serialize(PercentileSerializer.java:26) at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) at org.apache.kylin.engine.flink.FlinkCubingByLayer$1.map(FlinkCubingByLayer.java:234) at org.apache.kylin.engine.flink.FlinkCubingByLayer$1.map(FlinkCubingByLayer.java:218) at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:103) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704) at java.lang.Thread.run(Thread.java:745)

So we should provide a customized serializer for PercentileCounter.

Issue Links

• links to: GitHub Pull Request #613