Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-17480

New consumer commit all consumed should retrieve offsets in background thread

    XMLWordPrintableJSON

Details

    Description

      When committing all consumed offsets (sync, async, or on close), the new consumer retrieves the offsets from subscriptionState.allConsumed() in the app thread (and generates events containing those offsets, that are sent to the background thread to be processed). We should review this to consider retrieving the allConsumed in the background when processing the events, to avoid inconsistencies given that the subscription state could be modified in the background thread since the moment the allConsumed was retrieved in the app thread.

      These are the places in the app thread that currently retrieve the allConsumed to pass it into the events. 

      We could consider to keep the same CommitSync and CommitAsync events, but with the option of not taking offsets as input (maybe a flag for commitAllConsumed). We could also create new events for the allConsumed alternatives (I lean towards keeping just CommitSync and CommitAsync but open to whatever seems clearer)

      Attachments

        Issue Links

          Activity

            People

              yangpoan PoAn Yang
              lianetm Lianet Magrans
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: