Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1726 Scalable streaming cubing
  3. KYLIN-1744

Separate concepts of source offset and date range on cube segments

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.5.3
    • Component/s: None
    • Labels:
      None

      Description

      Previously date range is used to 1) cut data from source; 2) mark min/max data time for segment pruning. However for streaming source, these two concepts are separate. E.g. offset is used to cut data from Kafka; and the min/max data time of segments can overlap due to late coming records.

      Will add two more attributes in CubeSegment, sourceOffsetStart and sourceOffsetEnd. To be backward compatible, when the two attributes are missing (equals to 0), dateRangeStart and dateRangeEnd will serve as source offsets.

        Attachments

          Activity

            People

            • Assignee:
              liyang.gmt8@gmail.com liyang
              Reporter:
              liyang.gmt8@gmail.com liyang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: