Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13498

Reduce Kafka producer startup time by aborting transactions in parallel

    XMLWordPrintableJSON

Details

    Description

      When a Flink job with a Kafka producer starts up without previous state, it currently starts 5 * kafkaPoolSize number of Kafka producers (per sink instance) to abort potentially existing transactions from a first run without a completed snapshot.
      Apparently, this is quite slow and it is also done sequentially. Until there is a better way of aborting these transactions with Kafka, we could do this in parallel quite easily and at least make use of lingering CPU resources.

      Attachments

        Issue Links

          Activity

            People

              nkruber Nico Kruber
              nkruber Nico Kruber
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m