Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2331

Large TMP files created and never closed

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.4.0
    • None
    • File Channel
    • None

    Description

      We are currently writing files to Hadoop partitioned by year, month, day via Flume. File rollovers are done every 5 minutes. Recently, we noticed that this file rollover stops happening sometime during the day and that further data is written to an open TMP file. Because there are no further file rollovers, this TMP file becomes very large. At the end of the day, the TMP is not closed also and Flume goes to the next day, creating new files.

      We use a "." prefix to prevent Hive from complaining of the open TMP file. Because of this issue where the TMP file is never closed, the file remains hidden to Hive even after the day ends.

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            kkumar Krishna Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment