Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2331

Large TMP files created and never closed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: File Channel
    • Labels:
      None

      Description

      We are currently writing files to Hadoop partitioned by year, month, day via Flume. File rollovers are done every 5 minutes. Recently, we noticed that this file rollover stops happening sometime during the day and that further data is written to an open TMP file. Because there are no further file rollovers, this TMP file becomes very large. At the end of the day, the TMP is not closed also and Flume goes to the next day, creating new files.

      We use a "." prefix to prevent Hive from complaining of the open TMP file. Because of this issue where the TMP file is never closed, the file remains hidden to Hive even after the day ends.

        Attachments

        1. flumelog.zip
          5.45 MB
          Krishna Kumar

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kkumar Krishna Kumar
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: