[FLUME-2380] Encrypted files from spooldir to S3 using file channel

• Type: Question
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Component/s: File Channel

Environment

Linux \ S3

• Assignee: Unassigned
• Reporter: MX
• Created: 02/May/14 23:02
• Updated: 03/May/14 05:11

Description

0down votefavorite

I am a newbie to flume tring to move data from spooldir to S3, I am successfully able to move the data and roll the files into a bucket. I need to encrypt the data loaded into S3. I tried the following configuration in flume , log files tells me that "Encryption is enabled" but the files are written in S3 without encryption. Can someone tell me if flume encryption works with the config?

flume.conf
-----------
agent.sources = eventlog
agent.channels = file_channel
agent.sinks = s3sink

1. Define / Configure source
   agent.sources.eventlog.type = spooldir
   agent.sources.eventlog.spoolDir = /test/data
   agent.sources.eventlog.restart = true
   agent.sources.eventlog.batchSize = 1000
   #agent.sources.eventlog.type = seq

1. HDFS sinks
   agent.sinks.s3sink.type = hdfs
   agent.sinks.s3sink.hdfs.path = s3n://<ACCESSKEY>:<SECRETKEY>@<BUCKET>
   agent.sinks.s3sink.hdfs.fileType = DataStream
   agent.sinks.s3sink.hdfs.filePrefix = event
   agent.sinks.s3sink.hdfs.fileSuffix = .log
   agent.sinks.s3sink.hdfs.writeFormat = Text
   agent.sinks.s3sink.hdfs.batchSize = 1000
   agent.sinks.s3sink.hdfs.rollSize = 10485760
   agent.sinks.s3sink.hdfs.rollCount = 0
   agent.sinks.s3sink.hdfs.rollInterval = 60
   agent.sinks.s3sink.hdfs.threadsPoolSize = 100

1. Use a channel which buffers events in memory

agent.channels.file_channel.type = file
#25GB
agent.channels.fileChannel.maxFileSize = 25000000000
agent.channels.file_channel.checkpointDir = /var/flume/checkpoint
agent.channels.file_channel.dataDirs = /<directory>/data
agent.channels.file_channel.capacity = 50000000
agent.channels.file_channel.transactionCapacity = 10000

agent.channels.file_channel.encryption.activeKey = key-1
agent.channels.file_channel.encryption.cipherProvider = AESCTRNOPADDING
#agent.channels.file_channel.encryption.keyProvider = key-provider-1
agent.channels.file_channel.encryption.keyProvider = JCEKSFILE
agent.channels.file_channel.encryption.keyProvider.keys = key-1
agent.channels.file_channel.encryption.keyProvider.keyStoreFile = /<directory>/my.keystore
agent.channels.file_channel.encryption.keyProvider.keyStorePasswordFile = /<directory>/my.keystore.password

1. Bind the source and sink to the channel
   agent.sources.eventlog.channels = file_channel
   agent.sinks.s3sink.channel = file_channel