Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2495

Kafka Source may miss events when channel is not available

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.6.0
    • Sinks+Sources
    • None

    Description

      Because the Kafka consumer itself tracks offsets, and we don't restart the consumer when we get channel errors, the consumer will skip messages that we couldn't write to channel, even though we did not advance offset in Zookeeper.

      Attachments

        1. FLUME-2495.0.patch
          6 kB
          Gwen Shapira
        2. FLUME-2495.1.patch
          7 kB
          Gwen Shapira
        3. FLUME-2495.2.patch
          8 kB
          Gwen Shapira

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gwenshap Gwen Shapira
            gwenshap Gwen Shapira
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment