Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1318

Graceful shutdown and drain of flume agent

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Channel, Sinks+Sources
    • None

    Description

      It appears from our test that if i go and shutdown the agent which uses memory channel, there is no clean way of throttling incoming requests to source, draining events from memory channel and do clean graceful shutdown of an agent. We don't have a visibility of how much data(events) in memory channel are getting dropped when shutdown happens.

      It would be good if we add a method to throttle incoming requests to source, drain events from memory channel and then shutdown sink/channel/source. (We can certainly add counters/metrics as part of channel monitoring). I think file-channel covers by adding it to WAL.

      HBase RegionServer shutdown handler does the some drain process (it syncs the edits in Write-Ahead-Log and notifies the master for regions move/assignment)

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            inder Inder SIngh
            mubarakseyed Mubarak Seyed

            Dates

              Created:
              Updated:

              Slack

                Issue deployment