Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4500

Cassandra sink can lose messages

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.4.0
    • Component/s: Cassandra Connector
    • Labels:
      None

      Description

      The problem is the same as I pointed out with the Kafka producer sink (FLINK-4027). The CassandraTupleSink's send() and CassandraPojoSink's send() both send data asynchronously to Cassandra and record whether an error occurs via a future callback. But CassandraSinkBase does not implement Checkpointed, so it can't stop checkpoint from happening even though the are Cassandra queries in flight from the checkpoint that may fail. If they do fail, they would subsequently not be replayed when the job recovered, and would thus be lost.

      In addition,
      CassandraSinkBase's close should check whether there is a pending exception and throw it, rather than silently close. It should also wait for any pending async queries to complete and check their status before closing.

        Attachments

          Activity

            People

            • Assignee:
              mcfongtw Michael Fong
              Reporter:
              elevy Elias Levy
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: