[STORM-3393] OffsetManager doesn't recover after missing offsets

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.1.0, 1.1.3
• Component/s: storm-kafka-client
• Assignee: Unassigned
• Reporter: Greg Chatham
• Created: 16/May/19 20:48
• Updated: 21/May/19 16:11
• Resolved: 21/May/19 16:11

Description

When missing offsets are encountered, but a committable offset exists after the missing offset, the condition is detected and logged but not properly processed.  You will see three log messages in this case:

Processed non-sequential offset.  The earliest uncommitted offset is no longer part of the topic.  Missing offset: [{}], Processed: [{}]
...
Found committable offset: [{}] after missing offset: [{}], skipping to the committable offset
...
Topic-partition [{}] has no offsets ready to be committed

However, this is not the proper handling.  While a committable offset has been found, the found flag is not set to true (resulting in the 3rd log message).

The fix is to add a found=true within this logic:

In OffsetManager.java

if (nextEmittedOffset != null && currOffset == nextEmittedOffset) {
                        LOG.debug("Found committable offset: [{}] after missing offset: [{}], skipping to the committable offset",
                            currOffset, nextCommitOffset);
                        nextCommitOffset = currOffset + 1;
                        found = true;       //  ADD THIS LINE TO FIX THIS BUG
                    }

Because of this bug, offsets are not committed properly.

Issue Links

• Is contained by: STORM-3301 The KafkaSpout can in some cases still replay tuples that were already committed