Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3341

Taildir source may cause file handle leak and data duplication

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 1.8.0, 1.9.0
    • 1.8.0
    • Sinks+Sources
    • None
    • Flume 1.8.0 is fixed
    • Patch

    Description

      As is described in Flume-3342, renaming a file may cause data duplication. But moreover, if the original file was opened by flume before and not closed yet, flume reopens it while not freeing its original file handles. And then I delete the new file, but after the idle time, flume just closes the new file handle and forgets to close its original file handles, leading to a handle leak. It can be found by "lsof | grep "deleted".

      This bug is triggered by using  Log4j to roll log files. To avoid losing data when rolling, I use regex to include both current log file and old log files in the taildir path.

       

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            Edricming Ming Zhekai

            Dates

              Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 120h
              120h
              Remaining:
              Remaining Estimate - 119h 50m
              119h 50m
              Logged:
              Remaining Estimate - 119h 50m
              10m

              Issue deployment