Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2241

Spooling Directory Source doesn't handle 2 byte UTF-8 encoded characters correctly



    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Environment:

      Debian 6.0.5


      I have a flume agent set up with a spooling directory source sinking data to cassandra.

      I'm collecting web data writing a line in the log file for each request then once the log file has been rotated is dropped into the spooling directory ready for flume to start processing it. All data is valid json as its validated prior to it being written to the log file.

      Sending a mixture of different sized requests from 9-15k seems fine. Generated a log file of over 400Mb and it all sinked correctly.

      I'm currently logging a 19k request and this is when things start to break. It only gets as far as 1800th request in the file and the next one is truncated.

      Changed the sink to a file-roll sink and it only gets as far as 29Mb

      I have profiled it and it's not running out of memory. I want to know if there are any limitations on the spooling directory source.

      Has anyone tried dropping a file with similarly large requests and experienced a similar issue.

      Any pointers would be greatly appreciated. My flume config is as follows

      orion.sources = spoolDir
      orion.channels = fileChannel
      orion.sinks= cassandra
      orion.channels.fileChannel.type = file
      orion.channels.fileChannel.capacity = 1000000
      orion.channels.fileChannel.transactionCapacity = 100
      orion.channels.fileChannel.keep-alive = 60
      orion.channels.fileChannel.write-timeout = 60
      orion.sinks.cassandra.type = com.btoddb.flume.sinks.cassandra.CassandraSink
      orion.sinks.cassandra.hosts = <cluster node ip>
      orion.sinks.cassandra.cluster_name = fake_cluster
      orion.sinks.cassandra.port = 9160
      orion.sinks.cassandra.keyspace-name = Keysp
      orion.sinks.cassandra.records-colfam = <table>
      orion.sources.spoolDir.type = spooldir
      orion.sources.spoolDir.spoolDir = /var/log/orion/flumeSpooling
      orion.sources.spoolDir.deserializer = LINE
      orion.sources.spoolDir.inputCharset = UTF-8
      orion.sources.spoolDir.deserializer.maxLineLength = 20000000
      orion.sources.spoolDir.deletePolicy = never
      orion.sources.spoolDir.batchSize = 100
      orion.sources.spoolDir.interceptors = addSrc addHost addTimestamp addUUID
      orion.sources.spoolDir.interceptors.addSrc.type = regex_extractor
      orion.sources.spoolDir.interceptors.addSrc.regex = \"service\"\:\"([^"]*)
      orion.sources.spoolDir.interceptors.addSrc.serializers = s1
      orion.sources.spoolDir.interceptors.addSrc.serializers.s1.name = src
      orion.sources.spoolDir.interceptors.addUUID.type = regex_extractor
      orion.sources.spoolDir.interceptors.addUUID.regex = \"uuid\"\:\"([^"]*)
      orion.sources.spoolDir.interceptors.addUUID.serializers = s1
      orion.sources.spoolDir.interceptors.addUUID.serializers.s1.name = key
      orion.sources.spoolDir.interceptors.addHost.type = org.apache.flume.interceptor.HostInterceptor$Builder
      orion.sources.spoolDir.interceptors.addHost.preserveExisting = false
      orion.sources.spoolDir.interceptors.addHost.useIP = true
      orion.sources.spoolDir.interceptors.addHost.hostHeader = host
      orion.sources.spoolDir.interceptors.addTimestamp.type = regex_extractor
      orion.sources.spoolDir.interceptors.addTimestamp.regex = \"timestamp\"\:\"([^"]*)
      orion.sources.spoolDir.interceptors.addTimestamp.serializers = s1
      orion.sources.spoolDir.interceptors.addTimestamp.serializers.s1.name = timestamp
      orion.sources.spoolDir.channels = fileChannel
      orion.sinks.cassandra.channel = fileChannel

      Is this potentially a bug?.. If not tried can someone try to recreate - I hope the same error would occur.

      Dont hesitate to contact me for further info.



          Issue Links



              • Assignee:
                viktort Viktor Trako
              • Votes:
                0 Vote for this issue
                2 Start watching this issue


                • Created: