Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1810

Add IP Filtering / Whitelists-Blacklists

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: core, network, security
    • Labels:
      None

      Description

      While longer-term goals of security in Kafka are on the roadmap there exists some value for the ability to restrict connection to Kafka brokers based on IP address. This is not intended as a replacement for security but more of a precaution against misconfiguration and to provide some level of control to Kafka administrators about who is reading/writing to their cluster.

      1) In some organizations software administration vs o/s systems administration and network administration is disjointed and not well choreographed. Providing software administrators the ability to configure their platform relatively independently (after initial configuration) from Systems administrators is desirable.
      2) Configuration and deployment is sometimes error prone and there are situations when test environments could erroneously read/write to production environments
      3) An additional precaution against reading sensitive data is typically welcomed in most large enterprise deployments.

      1. KAFKA-1810_2015-01-15_19:47:14.patch
        49 kB
        Jeff Holoman
      2. KAFKA-1810_2015-03-15_01:13:12.patch
        37 kB
        Jeff Holoman
      3. KAFKA-1810.patch
        33 kB
        Jeff Holoman

        Issue Links

          Activity

          Hide
          jholoman Jeff Holoman added a comment -

          There was a discussion in KAFKA-1512 regarding the capability for limiting connections by setting max connections to 0. This gives a cleaner way to perform similar functionality

          Show
          jholoman Jeff Holoman added a comment - There was a discussion in KAFKA-1512 regarding the capability for limiting connections by setting max connections to 0. This gives a cleaner way to perform similar functionality
          Hide
          nehanarkhede Neha Narkhede added a comment -

          Jeff Holoman Not sure I understood what you are proposing. Can you be more specific about the changes you are proposing?

          Show
          nehanarkhede Neha Narkhede added a comment - Jeff Holoman Not sure I understood what you are proposing. Can you be more specific about the changes you are proposing?
          Hide
          jholoman Jeff Holoman added a comment -

          Neha Narkhede Sure no problem. I had a request to provide the ability to specify a range of IP addresses to either include or exclude. I was thinking the easiest way would be to specify IP addresses in CIDR notation and include them in the server.properties such as 192.168.2.0/24:allow, 192.168.1.0/16:deny. This would allow an administrator to accept/deny connections based on ip ranges. Does that clarify?

          Show
          jholoman Jeff Holoman added a comment - Neha Narkhede Sure no problem. I had a request to provide the ability to specify a range of IP addresses to either include or exclude. I was thinking the easiest way would be to specify IP addresses in CIDR notation and include them in the server.properties such as 192.168.2.0/24:allow, 192.168.1.0/16:deny. This would allow an administrator to accept/deny connections based on ip ranges. Does that clarify?
          Hide
          bosco Don Bosco Durai added a comment -

          Jeff Holoman, there is a JIRA for supporting authorization. https://issues.apache.org/jira/browse/KAFKA-1688. The current plan is to support user, group and IP. https://cwiki.apache.org/confluence/display/KAFKA/Security

          Would it address your issue?

          Show
          bosco Don Bosco Durai added a comment - Jeff Holoman , there is a JIRA for supporting authorization. https://issues.apache.org/jira/browse/KAFKA-1688 . The current plan is to support user, group and IP. https://cwiki.apache.org/confluence/display/KAFKA/Security Would it address your issue?
          Hide
          jholoman Jeff Holoman added a comment - - edited

          Don Bosco Durai ,

          I did note the broader security-related JIRA and have been following it somewhat closely. I'm keenly interested in this topic generally. I think this proposal makes sense for a number of reasons

          1) While this feature is certainly related to security its intent is really more to insulate against configuration issues and give software administrators the ability manage incoming connections by network range. For example, make sure my QA isn't writing to Prod.
          2) This is a relatively simple change that doesn't require dependencies on a much larger security initiative.
          3) There's no reason why this feature couldn't be worked into the upcoming permissions manager.
          4) The configuration is very minor, a list / range of IP addresses to allow/deny

          Essentially this would cover the following scenarios
          1) Allow any incoming connections but deny certain IP blocks
          2) The inverse of the above
          3) Some combination of the two, where certain subnets were allowed/denied, e.g. allow all traffic from 192.168.2.0/12 but deny 192.168.2.0/28.

          This concept is probably most related to IPTables or AWS security groups… What I mean here, we have other more sophisticated methods of authorizing access in other systems but the ability to filter simple network requests from a given IP range I think provides a valuable tool in the administrator's kit. As such I think this would provide this particular capability to restrict access in the near term until a more robust implementation is completed, or in conjunction with that initiative. As far as I know, the rollout for 1688 is TBD.

          Show
          jholoman Jeff Holoman added a comment - - edited Don Bosco Durai , I did note the broader security-related JIRA and have been following it somewhat closely. I'm keenly interested in this topic generally. I think this proposal makes sense for a number of reasons 1) While this feature is certainly related to security its intent is really more to insulate against configuration issues and give software administrators the ability manage incoming connections by network range. For example, make sure my QA isn't writing to Prod. 2) This is a relatively simple change that doesn't require dependencies on a much larger security initiative. 3) There's no reason why this feature couldn't be worked into the upcoming permissions manager. 4) The configuration is very minor, a list / range of IP addresses to allow/deny Essentially this would cover the following scenarios 1) Allow any incoming connections but deny certain IP blocks 2) The inverse of the above 3) Some combination of the two, where certain subnets were allowed/denied, e.g. allow all traffic from 192.168.2.0/12 but deny 192.168.2.0/28. This concept is probably most related to IPTables or AWS security groups… What I mean here, we have other more sophisticated methods of authorizing access in other systems but the ability to filter simple network requests from a given IP range I think provides a valuable tool in the administrator's kit. As such I think this would provide this particular capability to restrict access in the near term until a more robust implementation is completed, or in conjunction with that initiative. As far as I know, the rollout for 1688 is TBD.
          Hide
          joestein Joe Stein added a comment -

          +1 I like this approach because you can better manage the brokers to have resilience in their network environment for what hosts can connect to them. This is an implementation of what KAFKA-1688 will be layering and making pluggable. I also see overlap with https://issues.apache.org/jira/browse/KAFKA-1786 and might be a good place to start building that out too.

          Show
          joestein Joe Stein added a comment - +1 I like this approach because you can better manage the brokers to have resilience in their network environment for what hosts can connect to them. This is an implementation of what KAFKA-1688 will be layering and making pluggable. I also see overlap with https://issues.apache.org/jira/browse/KAFKA-1786 and might be a good place to start building that out too.
          Hide
          jholoman Jeff Holoman added a comment -

          Created reviewboard https://reviews.apache.org/r/29714/diff/
          against branch origin/trunk

          Show
          jholoman Jeff Holoman added a comment - Created reviewboard https://reviews.apache.org/r/29714/diff/ against branch origin/trunk
          Hide
          jholoman Jeff Holoman added a comment -

          This patch is a first pass at implementing IP Filtering logic. It requires defining two additional properties:
          security.ip.filter.rule.type
          security.ip.filter.list.

          The list of IP's are specified in CIDR notation: http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing. This implementation supports whitelisting ("allow" config value) or blacklisting ("deny"), mutually exclusive. The parameters are passed to socket server and validated upon startup.(I'd like to move most of the validation logic per KAFKA-1845). An exception is thrown and the server shutdown in the event of misconfiguration of these parameters. The check against the list is in the Acceptor thread, and if the rule check fails, the socket is closed. There are a lot of tests included in the patch but if there are suggestions for more please let me know.

          Show
          jholoman Jeff Holoman added a comment - This patch is a first pass at implementing IP Filtering logic. It requires defining two additional properties: security.ip.filter.rule.type security.ip.filter.list. The list of IP's are specified in CIDR notation: http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing . This implementation supports whitelisting ("allow" config value) or blacklisting ("deny"), mutually exclusive. The parameters are passed to socket server and validated upon startup.(I'd like to move most of the validation logic per KAFKA-1845 ). An exception is thrown and the server shutdown in the event of misconfiguration of these parameters. The check against the list is in the Acceptor thread, and if the rule check fails, the socket is closed. There are a lot of tests included in the patch but if there are suggestions for more please let me know.
          Hide
          jholoman Jeff Holoman added a comment -

          Updated reviewboard https://reviews.apache.org/r/29714/diff/
          against branch origin/trunk

          Show
          jholoman Jeff Holoman added a comment - Updated reviewboard https://reviews.apache.org/r/29714/diff/ against branch origin/trunk
          Hide
          jholoman Jeff Holoman added a comment -

          The current plan is to rework the configuration portion of this patch once KAFKA-1845 is committed (ConfigDef)

          Show
          jholoman Jeff Holoman added a comment - The current plan is to rework the configuration portion of this patch once KAFKA-1845 is committed (ConfigDef)
          Hide
          tongli Tong Li added a comment - - edited

          rather than add specific security measures, can we add some kind of plugin point so that any plugins can be configured to do that type of work. Either it is a IP filter or certificate filter or basic authentication filter we can simply enable these plugins according to our own needs. This way, kafka only provide the plugin point, nothing else, how the plugin gets developed , performs, are not really the concern of the kafka community, we can have a clear separation of concerns. This has been done in many other successful projects, new to kafka, just saying we can do some thing like middle ware (in python term) or servlet filter in java world. The point of doing this is to have the security measure become a configuration matter. One can choose any available plugins appropriate for their own purposes by changing configurations.

          Show
          tongli Tong Li added a comment - - edited rather than add specific security measures, can we add some kind of plugin point so that any plugins can be configured to do that type of work. Either it is a IP filter or certificate filter or basic authentication filter we can simply enable these plugins according to our own needs. This way, kafka only provide the plugin point, nothing else, how the plugin gets developed , performs, are not really the concern of the kafka community, we can have a clear separation of concerns. This has been done in many other successful projects, new to kafka, just saying we can do some thing like middle ware (in python term) or servlet filter in java world. The point of doing this is to have the security measure become a configuration matter. One can choose any available plugins appropriate for their own purposes by changing configurations.
          Hide
          jholoman Jeff Holoman added a comment -

          Updated reviewboard https://reviews.apache.org/r/29714/diff/
          against branch origin/trunk

          Show
          jholoman Jeff Holoman added a comment - Updated reviewboard https://reviews.apache.org/r/29714/diff/ against branch origin/trunk

            People

            • Assignee:
              jholoman Jeff Holoman
              Reporter:
              jholoman Jeff Holoman
              Reviewer:
              Gwen Shapira
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development