Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1350

HDFS file handle not closed properly when date bucketing

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.1.0, 1.2.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      With configuration:

      agent.sinks.hdfs-cafe-access.type = hdfs
      agent.sinks.hdfs-cafe-access.hdfs.path = hdfs://nga/nga/apache/access/%y-%m-%d/
      agent.sinks.hdfs-cafe-access.hdfs.fileType = DataStream
      agent.sinks.hdfs-cafe-access.hdfs.filePrefix = cafe_access
      agent.sinks.hdfs-cafe-access.hdfs.rollInterval = 21600
      agent.sinks.hdfs-cafe-access.hdfs.rollSize = 10485760
      agent.sinks.hdfs-cafe-access.hdfs.rollCount = 0
      agent.sinks.hdfs-cafe-access.hdfs.txnEventMax = 1000
      agent.sinks.hdfs-cafe-access.hdfs.batchSize = 1000
      #agent.sinks.hdfs-cafe-access.hdfs.codeC = snappy
      agent.sinks.hdfs-cafe-access.hdfs.hdfs.maxOpenFiles = 5000
      agent.sinks.hdfs-cafe-access.channel = memo-1

      When new directory is created previous file handle remains opened. rollInterval setting is used only with files in current date bucket.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                dex Robert Mroczkowski
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: