Using storm and storm-kafka 0.9.2-incubating
In the storm kafka spout the default for maxOffsetBehind is 100000
This default is too low and causes the kafka spout to start from the latest offset instead of the last committed offset without warning.
Producing the following log output from the storm worker processes
To fix this problem I ended up setting spout config in my topology like so
Why would the kafka spout skip to the latest offset if the current offset
is more then 100000 behind by default?
This seems like a bad default value, the spout literally skipped over
months of data without any warning.
Are the core contributors open to accepting a pull request that would set
the default to Long.MAX_VALUE?