Details
-
Question
-
Status: Closed
-
Minor
-
Resolution: Invalid
-
2.3.0
-
None
-
None
Description
We recently observed that the spark structured streaming generated duplicates in the output when reading from Kafka topic and storing the output to the S3 (and checkpointing in S3). We ran into this issue twice. This is not reproducible. Is there anyone has ever faced this kind of issue before? Is this because of S3 eventual consistency?