[KYLIN-4743] NPE when some kafka partition's offset is not exists in checkpoint

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v3.0.0, v3.1.0
• Fix Version/s: Future
• Component/s: Real-time Streaming
• Environment: Centos 7.4.1 ; hbase 1.2.4 ; hive 2.0.1 ; hadoop 2.7.2
• Flags: Patch
• Assignee: GuKe
• Reporter: GuKe
• Due: 17/Sep/20
• Created: 03/Sep/20 07:41
• Updated: 21/Sep/20 03:35
• Resolved: 21/Sep/20 03:35

Description

We have used streaming cube to statistics data from Kafka.
If the data partitions of kafka topic is imbalanced (one or more partition is empty),it will cause the cube's checkpoint contains one or more partition's offset is empty.
For some reason the receiver node be restarted,it will fail to initialization kafka connector because offset is null.

Issue Links

• links to: https://github.com/apache/kylin/pull/1392