System test StreamsEosTest.test_failure_and_recovery failed due to a wrongly computed aggregation under exactly-once (EOS). The specific error is:
That means, the sum computed by the Streams app seems to be wrong for key 6069. I checked the dumps of the log segments of the input topic partition (attached: data-1.txt) and indeed two input records are not considered in the sum. With those two missed records the sum would be correct. More concretely, the input values for key 6069 are:
The sum of this values is 17269 as stated in the exception above as expected sum. If you subtract values 3 and 4, i.e., 5340 and 1231 from 17269, you get 10698 , which is the actual sum in the exception above. Somehow those two values are missing.
In the log dump of the output topic partition (attached: sum-1.txt), the sum is correct until the 4th value 1231 , i.e. 15968, then it is overwritten with 10698.
In the log dump of the changelog topic of the state store that stores the sum (attached: 7-changelog-1.txt), the sum is also overwritten as in the output topic.
I attached the logs of the three Streams instances involved.