[SAMZA-2088] Use startpoint to calculate the starting offset of a SSP in OffsetManager.

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 25/Jan/19 23:15
• Updated: 15/May/19 23:12
• Resolved: 15/May/19 23:12

Description

TaskInstance receives input messages from the AsyncRunLoop(i.e, event loop that coordinates the input message dispatch to the tasks within a SamzaContainer) and routes them to the user-defined task implementation.

Starting offset for a input SystemStreamPartition is calculated in samza as one offset ahead of the previous checkpointed offset of the SystemStreamPartition.

Currently, TaskInstance is instantiated with the starting offsets as part of AsycnRunLoop initialization. TaskInstance does not route the message of a SSP to the user-defined task implementation when the offset of received message is lower than the starting offset of SSP.

SAMZA-1985 introduced the startpoint abstraction, where during AsyncRunLoop initialization, the idea is to move the input consumer to an offset defined by the startpoint.

However, moving the consumer to an offset back alone does not suffice. For instance, if the check-pointed offset is 100 for a SSP and the startpoint intends to move the consumer back to offset 0. Even if we successfully moved the consumer instance to offset 0, the messages from offset 0 to 99 will not be routed by TaskInstance to the user defined task implementation.

Issue Links

• is a child of: SAMZA-1983 SEP-18 : Startpoints implementation in the core framework