Kim and I have tried a first go at a patch for this, but it's not working and we're running low on ideas (and inspiration) =[
To get our patch working, you need to set flume.secure.transport to true in flume-site.xml. You also need to set up a Java keystore and truststore on any masters and nodes (we're testing with a master process and node process running on the same VM). You need to reference the keystore and truststore in ./bin/flume or else the JVM will throw exceptions. It's not too bad, just follow these instructions: http://gist.github.com/536015
We tested with this setup configuring...
agent1: tail("/path/to/file") | agentDFOSink("ip.of.machine")
collector1: collectorSource | text("/path/to/output_file")
The master process runs, and so does the node process. It also appears that the DFO stuff works, but the messages never make it to their destination. If we turn off the flume.secure.transport switch, everything seems to work. We suspect it is something with the TTransport type (http://github.com/aguynamedben/flume/commit/e5dea42df2e5235308664b492df6377768139265#L0L87) or TSaneServerSocket.java, its binding or lazy port opening. Errors we saw along the way included TSocket's "Socket already connected." Can somebody familiar with Thrift, ThriftMasterRPC.java, TSaneServerSocket.java, or the server thread pool stuff take a look? We've spun our heads multiple times. Thanks!