Major Description
Because Connect started with a worker-wide model of data formats, we currently allocate a single Converter per worker and only allocate an independent one when the user overrides the converter.
This can lead to performance problems when the worker-level default converter is used by a large number of tasks because converters need to be threadsafe to support this model and they may spend a lot of time just on synchronization.
We could, instead, simply allocate one converter per task. There is some overhead involved, but generally it shouldn't be that large. For example, Confluent's Avro converters will each have their own schema cache and have to make their on calls to the schema registry API, but these are relatively small, likely inconsequential compared to any normal overhead we would already have for creating and managing each task.
Attachments
Issue Links
- links to
