[KYLIN-1744] Separate concepts of source offset and date range on cube segments

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v1.5.3
• Assignee: liyang
• Reporter: liyang
• Created: 28/May/16 01:07
• Updated: 28/Jul/16 06:48
• Resolved: 29/May/16 14:10

Description

Previously date range is used to 1) cut data from source; 2) mark min/max data time for segment pruning. However for streaming source, these two concepts are separate. E.g. offset is used to cut data from Kafka; and the min/max data time of segments can overlap due to late coming records.

Will add two more attributes in CubeSegment, sourceOffsetStart and sourceOffsetEnd. To be backward compatible, when the two attributes are missing (equals to 0), dateRangeStart and dateRangeEnd will serve as source offsets.