Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25132

KafkaSource cannot work with object-reusing DeserializationSchema

    XMLWordPrintableJSON

Details

    Description

      Currently Kafka source deserializes ConsumerRecords in split reader and puts them into the elementQueue, then task's main thread polls these records from the queue asynchronously. This mechanism cannot cooperate with DeserializationSchemas with object reuse: all records staying in the element queue points to the same object.

      A solution would be moving deserialization to RecordEmitter, which works in the task's main thread. 

      Notes that this issue actually effects all sources which do deserialization in split reader. 

      Attachments

        Issue Links

          Activity

            People

              renqs Qingsheng Ren
              renqs Qingsheng Ren
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: