Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.14.5, 1.15.3
-
None
-
None
-
None
Description
Currently if the interval between the arrival of the delete message and the insert/update message exceeds state ttl, the delete message was ignored incorrectly in `SinkUpsertMaterializer`. This will cause wrong result in corresponding sink table(dirty data left).
1. if state ttl is set to '10 hour', then the following delete message will be ignored (the '+I (1, a1)' will be left in sink table forever)
00:00:01 +I (1, a1) 10:00:02 -D (1, a1)
2. but another contrast case which will wrongly delete data in sink table if we send delete message when state staled
00:00:01 +I (1, a1) 00:00:02 +I (1, a2) 10:00:03 -D (1, a1)
compare the two choice of current implementation and eager deletion, the former will cause dirty data left, but the later will cause some data lost(seems the former is less harmful..)