Flume
  1. Flume
  2. FLUME-13

Secure transport from outside of a corporate firewall to within a corporate firewall

    Details

    • Type: New Feature New Feature
    • Status: Reopened
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: v0.9.4
    • Fix Version/s: v0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      I'm not sure of the exact feature gap here, but I'd like to have a Flume agent running on a server outside of our corporate firewall shoveling data to a Flume collector living inside of the corporate firewall, and I want to make sure the connection is secure.

        Issue Links

          Activity

          Disabled imported user created issue -
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Component/s Sinks+Sources [ 10041 ]
          Patrick Hunt made changes -
          Workflow jira [ 10028 ] flume-workflow [ 10155 ]
          Hide
          Jonathan Hsieh added a comment -

          "Sockets are used for communication between the Flume processes. This is not a problem
          when running the Flume processes behind a firewall, but if some of the agents are forward
          deployed, outside of the firewall surrounding a cluster, having the content streamed without
          encryption will be a problem. Encryption and authentication are a must."

          Show
          Jonathan Hsieh added a comment - "Sockets are used for communication between the Flume processes. This is not a problem when running the Flume processes behind a firewall, but if some of the agents are forward deployed, outside of the firewall surrounding a cluster, having the content streamed without encryption will be a problem. Encryption and authentication are a must."
          Hide
          Kim Vogt added a comment -

          The proposal is to add java support for TLS to thrift. In order to push this upstream to thrift, we may need to add support for every language, but since we just need the java support, how about just an extra class that flume maintains for secure sockets?

          Show
          Kim Vogt added a comment - The proposal is to add java support for TLS to thrift. In order to push this upstream to thrift, we may need to add support for every language, but since we just need the java support, how about just an extra class that flume maintains for secure sockets?
          Hide
          Disabled imported user added a comment -

          I agree that TLS looks like the way to go. It's a matter of implementing a custom TSecureSocket class that could eventually be integrated upstream. I'm not even sure if the Thrift folks would outright turn down such a contribution based solely on the fact that someone didn't show up with perfect implementations for all the languages they have... (here, I'm volunteering to do the PHP implementation for Thrift, that's already a second language done!)

          For Flume's purposes, a Java implementation would suffice, as the goal is to secure the transport between agents and collectors, and they're both Java. Anything else is outside the scope of Flume in my opinion. So if Thrift accepts a secure socket contribution, fine... if not, then it could stay a custom implementation that resides in Flume - still the best option in terms of long-term maintainability. TLS has all the necessary stuff figured out already. Rolling anything custom would be nightmarish.

          Show
          Disabled imported user added a comment - I agree that TLS looks like the way to go. It's a matter of implementing a custom TSecureSocket class that could eventually be integrated upstream. I'm not even sure if the Thrift folks would outright turn down such a contribution based solely on the fact that someone didn't show up with perfect implementations for all the languages they have... (here, I'm volunteering to do the PHP implementation for Thrift, that's already a second language done!) For Flume's purposes, a Java implementation would suffice, as the goal is to secure the transport between agents and collectors, and they're both Java. Anything else is outside the scope of Flume in my opinion. So if Thrift accepts a secure socket contribution, fine... if not, then it could stay a custom implementation that resides in Flume - still the best option in terms of long-term maintainability. TLS has all the necessary stuff figured out already. Rolling anything custom would be nightmarish.
          Hide
          Disabled imported user added a comment -

          [02:42] kimsterv: TSocket in thrift can be initialized with a Socket so we can call
          [02:42] kimsterv: TTransport masterTransport = new TSocket(SSLSocketFactory.getDefault().createSocket(host, port));
          [02:42] kimsterv: in ThriftMasterRPC.java in flume
          [02:42] kimsterv: but I'm not sure if it's as easy for the non-blocking sockets

          Show
          Disabled imported user added a comment - [02:42] kimsterv: TSocket in thrift can be initialized with a Socket so we can call [02:42] kimsterv: TTransport masterTransport = new TSocket(SSLSocketFactory.getDefault().createSocket(host, port)); [02:42] kimsterv: in ThriftMasterRPC.java in flume [02:42] kimsterv: but I'm not sure if it's as easy for the non-blocking sockets
          Hide
          Disabled imported user added a comment -

          We also need to consider how to secure Avro connections. We will be moving towards Avro in the (near) future for all RPCs.

          Show
          Disabled imported user added a comment - We also need to consider how to secure Avro connections. We will be moving towards Avro in the (near) future for all RPCs.
          Hide
          Disabled imported user added a comment -

          Avro uses a java.nio.channels.SocketChannel, and it should be possible to hand it a secure socket or write a wrapper. Tomcat does something like that, for instance: http://tomcat.apache.org/tomcat-6.0-doc/api/org/apache/tomcat/util/net/SecureNioChannel.html

          Show
          Disabled imported user added a comment - Avro uses a java.nio.channels.SocketChannel, and it should be possible to hand it a secure socket or write a wrapper. Tomcat does something like that, for instance: http://tomcat.apache.org/tomcat-6.0-doc/api/org/apache/tomcat/util/net/SecureNioChannel.html
          Hide
          Patrick Hunt added a comment -

          The question in my mind is how is this configured? Seems like it should be in the logical configuration of a node, however it's "cross cutting" and we don't want to have to change particular source/sink to support. Rather it would be nice to have a concept like decorator(although current deco won't work here) where we can annotate a source/sink with additional attributes - "encrypt this sink with ssl" or somesuch. Encryption and authentication both seem issues to me. (certificate auth in the ssl case as one example). sasl?

          Show
          Patrick Hunt added a comment - The question in my mind is how is this configured? Seems like it should be in the logical configuration of a node, however it's "cross cutting" and we don't want to have to change particular source/sink to support. Rather it would be nice to have a concept like decorator(although current deco won't work here) where we can annotate a source/sink with additional attributes - "encrypt this sink with ssl" or somesuch. Encryption and authentication both seem issues to me. (certificate auth in the ssl case as one example). sasl?
          Hide
          Disabled imported user added a comment -

          For SSL, how about an installation-wide configuration variable in flume-conf.xml or flume-site.xml that enables or disables SSL connections for all Flume communication, including sources/sinks? If you enable this variable, it's up to you to implement SSL certificate authentication for your infrastructure, which isn't a huge deal if you infrastructure is automated with Puppet or Chef. Once this global variable is enabled, it allows you to go about using sources/sinks, reroute logging at will, etc, without worrying about the security details at the source/sink configuration level. If security configuration was necessary at the source/sink/decorator level, I'd be paranoid to use the web interface or Flume shell to change configuration dynamically in fear that me or somebody else would accidentally configure sensitive clear text to be sent over the wire. This violates the dynamic re-routing design principles of Flume.

          Show
          Disabled imported user added a comment - For SSL, how about an installation-wide configuration variable in flume-conf.xml or flume-site.xml that enables or disables SSL connections for all Flume communication, including sources/sinks? If you enable this variable, it's up to you to implement SSL certificate authentication for your infrastructure, which isn't a huge deal if you infrastructure is automated with Puppet or Chef. Once this global variable is enabled, it allows you to go about using sources/sinks, reroute logging at will, etc, without worrying about the security details at the source/sink configuration level. If security configuration was necessary at the source/sink/decorator level, I'd be paranoid to use the web interface or Flume shell to change configuration dynamically in fear that me or somebody else would accidentally configure sensitive clear text to be sent over the wire. This violates the dynamic re-routing design principles of Flume.
          Hide
          Disabled imported user added a comment -

          On second thought, I don't think a global SSL configuration variable would work because there are some sinks/sources that don't natively support SSL (like syslog?). Maybe there is a way it would enable/disable the main Flume-centric sinks/sources BESink, DFOSink, E2ESink, CollectorSink, CollectorSource.

          Show
          Disabled imported user added a comment - On second thought, I don't think a global SSL configuration variable would work because there are some sinks/sources that don't natively support SSL (like syslog?). Maybe there is a way it would enable/disable the main Flume-centric sinks/sources BESink, DFOSink, E2ESink, CollectorSink, CollectorSource.
          Hide
          Disabled imported user added a comment -

          And the gossip, with a warning when you configure a source/sink that doesn't support SSL.

          Show
          Disabled imported user added a comment - And the gossip, with a warning when you configure a source/sink that doesn't support SSL.
          Hide
          Disabled imported user added a comment -

          That would indeed not work, also because it would require all sources to send over SSL (which e.g. for Thrift sources means that all language implementations would need this capability, eww), which is not really the scope of this functionality.

          I think it's reasonable to assume that 99,975% of people will want to have agents in one location communicate with collectors in another location, over a potentially unsafe connection. That's where we need transport security. My local Scribe client can communicate with my local agent without SSL just fine.

          Show
          Disabled imported user added a comment - That would indeed not work, also because it would require all sources to send over SSL (which e.g. for Thrift sources means that all language implementations would need this capability, eww), which is not really the scope of this functionality. I think it's reasonable to assume that 99,975% of people will want to have agents in one location communicate with collectors in another location, over a potentially unsafe connection. That's where we need transport security. My local Scribe client can communicate with my local agent without SSL just fine.
          Hide
          Disabled imported user added a comment -

          Kim and I have tried a first go at a patch for this, but it's not working and we're running low on ideas (and inspiration) =[

          http://github.com/aguynamedben/flume/commit/e5dea42df2e5235308664b492df6377768139265

          To get our patch working, you need to set flume.secure.transport to true in flume-site.xml. You also need to set up a Java keystore and truststore on any masters and nodes (we're testing with a master process and node process running on the same VM). You need to reference the keystore and truststore in ./bin/flume or else the JVM will throw exceptions. It's not too bad, just follow these instructions: http://gist.github.com/536015

          We tested with this setup configuring...
          agent1: tail("/path/to/file") | agentDFOSink("ip.of.machine")
          collector1: collectorSource | text("/path/to/output_file")

          The master process runs, and so does the node process. It also appears that the DFO stuff works, but the messages never make it to their destination. If we turn off the flume.secure.transport switch, everything seems to work. We suspect it is something with the TTransport type (http://github.com/aguynamedben/flume/commit/e5dea42df2e5235308664b492df6377768139265#L0L87) or TSaneServerSocket.java, its binding or lazy port opening. Errors we saw along the way included TSocket's "Socket already connected." Can somebody familiar with Thrift, ThriftMasterRPC.java, TSaneServerSocket.java, or the server thread pool stuff take a look? We've spun our heads multiple times. Thanks!

          Show
          Disabled imported user added a comment - Kim and I have tried a first go at a patch for this, but it's not working and we're running low on ideas (and inspiration) =[ http://github.com/aguynamedben/flume/commit/e5dea42df2e5235308664b492df6377768139265 To get our patch working , you need to set flume.secure.transport to true in flume-site.xml. You also need to set up a Java keystore and truststore on any masters and nodes (we're testing with a master process and node process running on the same VM). You need to reference the keystore and truststore in ./bin/flume or else the JVM will throw exceptions. It's not too bad, just follow these instructions: http://gist.github.com/536015 We tested with this setup configuring... agent1: tail("/path/to/file") | agentDFOSink("ip.of.machine") collector1: collectorSource | text("/path/to/output_file") The master process runs, and so does the node process. It also appears that the DFO stuff works, but the messages never make it to their destination. If we turn off the flume.secure.transport switch, everything seems to work. We suspect it is something with the TTransport type ( http://github.com/aguynamedben/flume/commit/e5dea42df2e5235308664b492df6377768139265#L0L87 ) or TSaneServerSocket.java, its binding or lazy port opening. Errors we saw along the way included TSocket's "Socket already connected." Can somebody familiar with Thrift, ThriftMasterRPC.java, TSaneServerSocket.java, or the server thread pool stuff take a look? We've spun our heads multiple times. Thanks!
          Hide
          Kim Vogt added a comment -

          I think the problem now is that we didn't update the sinks/sources to be secure. The examples we were running use the ThriftEventSink and they're still using normal TSockets. I'm not sure if changing these sinks/sources is as trivial because of the "nonblocking" stuff. We also changed the TSaneServerSocket to create its socket with the port, inetaddress, and backlog instead of just the port.

          Show
          Kim Vogt added a comment - I think the problem now is that we didn't update the sinks/sources to be secure. The examples we were running use the ThriftEventSink and they're still using normal TSockets. I'm not sure if changing these sinks/sources is as trivial because of the "nonblocking" stuff. We also changed the TSaneServerSocket to create its socket with the port, inetaddress, and backlog instead of just the port.
          Hide
          Kim Vogt added a comment -

          I added secure sockets to the ThriftEventSink and now our test case works. I'll get these changes committed to Ben's branch.

          Show
          Kim Vogt added a comment - I added secure sockets to the ThriftEventSink and now our test case works. I'll get these changes committed to Ben's branch.
          Hide
          Kim Vogt added a comment - - edited

          We attempted a patch for this. All changes made can be viewed here:

          http://github.com/aguynamedben/flume/commit/e456e599a9a0c4cc394371c0461dde888a7f4b5e

          Several things to note:
          1. I didn't update the flume shell to use SSL sockets
          2. I only changed the ThriftEventSink.java, if other sinks are needed, they'll have to be modified
          3. In ThriftEventSink.java I didn't do anything fancy with the nonblocking IO. Thrift may handle this with TFramedTransport but I'm not sure.
          4. In ThriftMasterRPC.java, I picked a random number for the backlog for the socket creation
          5. Some notes on how we set up the certificates http://gist.github.com/539295

          We're not sure if we're going to use this, but maybe someone will find it helpful.

          Show
          Kim Vogt added a comment - - edited We attempted a patch for this. All changes made can be viewed here: http://github.com/aguynamedben/flume/commit/e456e599a9a0c4cc394371c0461dde888a7f4b5e Several things to note: 1. I didn't update the flume shell to use SSL sockets 2. I only changed the ThriftEventSink.java, if other sinks are needed, they'll have to be modified 3. In ThriftEventSink.java I didn't do anything fancy with the nonblocking IO. Thrift may handle this with TFramedTransport but I'm not sure. 4. In ThriftMasterRPC.java, I picked a random number for the backlog for the socket creation 5. Some notes on how we set up the certificates http://gist.github.com/539295 We're not sure if we're going to use this, but maybe someone will find it helpful.
          Hide
          Patrick Hunt added a comment -

          Hi Kim, it's really great that you are making this available. See this link for our contribution process:
          http://wiki.github.com/cloudera/flume/howtocontribute

          It's fairly straightforward - basically you need to attach the patch to a review on reviewboard. You also need a CCLA on file for us to be able to accept your code into the main repository.

          regards.

          Show
          Patrick Hunt added a comment - Hi Kim, it's really great that you are making this available. See this link for our contribution process: http://wiki.github.com/cloudera/flume/howtocontribute It's fairly straightforward - basically you need to attach the patch to a review on reviewboard. You also need a CCLA on file for us to be able to accept your code into the main repository. regards.
          Hide
          Disabled imported user added a comment -

          Kim and everyone else,

          great work! A few questions though:

          1) this is a global setting, right? Would it be possible and/or desirable to have per-flow security? No idea how to configure that on the flow side (new params? decorators? ...), but on the flume-conf.xml side, it could be simply named transports ("flume.secure.transport.foobar").
          2) is there a way to have the javax.ssl.* settings in flume-conf.xml, or is it one of these completely miserable global settings that can't be changed on a per-connection/-socket basis?
          3) shouldn't EventSink.Base have an abstract public void open(boolean secured) and a concrete implementation of public void open() that calls open(secured), which in turn could throw an exception in case the concrete sink impl didn't allow for transport security? Might help encourage people to implement secure transports, but I'm not sure...

          Show
          Disabled imported user added a comment - Kim and everyone else, great work! A few questions though: 1) this is a global setting, right? Would it be possible and/or desirable to have per-flow security? No idea how to configure that on the flow side (new params? decorators? ...), but on the flume-conf.xml side, it could be simply named transports ("flume.secure.transport.foobar"). 2) is there a way to have the javax.ssl.* settings in flume-conf.xml, or is it one of these completely miserable global settings that can't be changed on a per-connection/-socket basis? 3) shouldn't EventSink.Base have an abstract public void open(boolean secured) and a concrete implementation of public void open() that calls open(secured), which in turn could throw an exception in case the concrete sink impl didn't allow for transport security? Might help encourage people to implement secure transports, but I'm not sure...
          Hide
          Jonathan Hsieh added a comment -

          We'd like to get this feature into the next version of flume (0.9.2)

          Show
          Jonathan Hsieh added a comment - We'd like to get this feature into the next version of flume (0.9.2)
          Jonathan Hsieh made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Fix Version/s v0.9.2 [ 10022 ]
          Hide
          Kim Vogt added a comment -

          TSSLServerSocket and TSSLSocket implementation in Thrift:

          https://issues.apache.org/jira/browse/THRIFT-151

          Show
          Kim Vogt added a comment - TSSLServerSocket and TSSLSocket implementation in Thrift: https://issues.apache.org/jira/browse/THRIFT-151
          Hide
          Disabled imported user added a comment -

          Any updates on getting this feature into 0.9.2? I have a project that could really use it.

          Show
          Disabled imported user added a comment - Any updates on getting this feature into 0.9.2? I have a project that could really use it.
          Disabled imported user made changes -
          Status Open [ 1 ] Patch Available [ 10000 ]
          Disabled imported user made changes -
          Status Patch Available [ 10000 ] Open [ 1 ]
          Jonathan Hsieh made changes -
          Fix Version/s v0.9.3 [ 10040 ]
          Fix Version/s v0.9.2 [ 10022 ]
          Hide
          Jonathan Hsieh added a comment -

          This feature is going to miss the boat on 0.9.2, but will make 0.9.3. We will likely use add SASL thrift interface that is used in Hue. Also, this may be a vehicle for doign transport encryption in Avro.

          Show
          Jonathan Hsieh added a comment - This feature is going to miss the boat on 0.9.2, but will make 0.9.3. We will likely use add SASL thrift interface that is used in Hue. Also, this may be a vehicle for doign transport encryption in Avro.
          Jonathan Hsieh made changes -
          Fix Version/s v0.9.3 [ 10040 ]
          Fix Version/s v0.9.4 [ 10050 ]
          Hide
          Jonathan Hsieh added a comment -

          This is going to miss 0.9.3 – upgrades to the thrift library should make this easier. Need to do some lookups to see if the avro side has similar facilities. This will be a blocker for 0.9.4.

          Show
          Jonathan Hsieh added a comment - This is going to miss 0.9.3 – upgrades to the thrift library should make this easier. Need to do some lookups to see if the avro side has similar facilities. This will be a blocker for 0.9.4.
          Jonathan Hsieh made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          Assignee Bruce Mitchener [ brucem ]
          Jonathan Hsieh made changes -
          Fix Version/s v0.9.5 [ 10090 ]
          Fix Version/s v0.9.4 [ 10050 ]
          Mark Thomas made changes -
          Project Import Tue Aug 02 16:57:12 UTC 2011 [ 1312304232406 ]
          Hide
          Jonathan Hsieh added a comment -

          New features should not be blockers.

          Show
          Jonathan Hsieh added a comment - New features should not be blockers.
          Jonathan Hsieh made changes -
          Assignee Bruce Mitchener [ brucem ]
          Priority Blocker [ 1 ] Critical [ 2 ]
          Arvind Prabhakar made changes -
          Affects Version/s v0.9.4 [ 12317557 ]
          Mike Percy made changes -
          Link This issue relates to FLUME-997 [ FLUME-997 ]
          Mike Percy made changes -
          Link This issue relates to FLUME-994 [ FLUME-994 ]
          Hide
          Alexander Alten-Lorenz added a comment -

          deprecated, flumeNG (1.0x)

          Show
          Alexander Alten-Lorenz added a comment - deprecated, flumeNG (1.0x)
          Alexander Alten-Lorenz made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee Alexander Lorenz-Alten [ alo.alt ]
          Resolution Won't Fix [ 2 ]
          Alexander Alten-Lorenz made changes -
          Resolution Won't Fix [ 2 ]
          Status Resolved [ 5 ] Reopened [ 4 ]

            People

            • Assignee:
              Alexander Alten-Lorenz
              Reporter:
              Disabled imported user
            • Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development