[GOBBLIN-1470] Commit watermarks to state store for quiet topics with no data

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.15.0
• Fix Version/s: 0.17.0
• Component/s: gobblin-kafka
• Assignee: Sudarshan Vasudevan
• Reporter: Sudarshan Vasudevan
• Created: 12/Jun/21 01:04
• Updated: 14/Jun/23 04:13

Description

Currently, quiet topics which see no data during a flush interval will not commit watermark to state store. This can result in quiet topics not having any prior stats resulting in an excess number of workunits for such topics. The correct behavior should be to have watermarks with produce rates set to 0.