Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2380

Encrypted files from spooldir to S3 using file channel

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: File Channel
    • Labels:
      None
    • Environment:

      Linux \ S3

      Description

      0down votefavorite

      I am a newbie to flume tring to move data from spooldir to S3, I am successfully able to move the data and roll the files into a bucket. I need to encrypt the data loaded into S3. I tried the following configuration in flume , log files tells me that "Encryption is enabled" but the files are written in S3 without encryption. Can someone tell me if flume encryption works with the config?

      flume.conf
      -----------
      agent.sources = eventlog
      agent.channels = file_channel
      agent.sinks = s3sink

      1. Define / Configure source
        agent.sources.eventlog.type = spooldir
        agent.sources.eventlog.spoolDir = /test/data
        agent.sources.eventlog.restart = true
        agent.sources.eventlog.batchSize = 1000
        #agent.sources.eventlog.type = seq
      1. HDFS sinks
        agent.sinks.s3sink.type = hdfs
        agent.sinks.s3sink.hdfs.path = s3n://<ACCESSKEY>:<SECRETKEY>@<BUCKET>
        agent.sinks.s3sink.hdfs.fileType = DataStream
        agent.sinks.s3sink.hdfs.filePrefix = event
        agent.sinks.s3sink.hdfs.fileSuffix = .log
        agent.sinks.s3sink.hdfs.writeFormat = Text
        agent.sinks.s3sink.hdfs.batchSize = 1000
        agent.sinks.s3sink.hdfs.rollSize = 10485760
        agent.sinks.s3sink.hdfs.rollCount = 0
        agent.sinks.s3sink.hdfs.rollInterval = 60
        agent.sinks.s3sink.hdfs.threadsPoolSize = 100
      1. Use a channel which buffers events in memory

      agent.channels.file_channel.type = file
      #25GB
      agent.channels.fileChannel.maxFileSize = 25000000000
      agent.channels.file_channel.checkpointDir = /var/flume/checkpoint
      agent.channels.file_channel.dataDirs = /<directory>/data
      agent.channels.file_channel.capacity = 50000000
      agent.channels.file_channel.transactionCapacity = 10000

      agent.channels.file_channel.encryption.activeKey = key-1
      agent.channels.file_channel.encryption.cipherProvider = AESCTRNOPADDING
      #agent.channels.file_channel.encryption.keyProvider = key-provider-1
      agent.channels.file_channel.encryption.keyProvider = JCEKSFILE
      agent.channels.file_channel.encryption.keyProvider.keys = key-1
      agent.channels.file_channel.encryption.keyProvider.keyStoreFile = /<directory>/my.keystore
      agent.channels.file_channel.encryption.keyProvider.keyStorePasswordFile = /<directory>/my.keystore.password

      1. Bind the source and sink to the channel
        agent.sources.eventlog.channels = file_channel
        agent.sinks.s3sink.channel = file_channel

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              micxav MX
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: