Cassandra
  1. Cassandra
  2. CASSANDRA-2274

Restrict Cassandra cluster node joins to a list of named hosts

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not a Problem
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None
    • Environment:

      All

      Description

      Because firewalls and employees are not infallible it would be nice to restrict the ability of any node to join a cluster to a list of named hosts in the configuration so that someone would be unable to start a node and replicate all the data locally. I understand that in order to do this the person must know the seed servers and the cluster name and to extract the data they will need a userid and password but another level of security would be to force them to execute any brute force attack from a locked down server instead of replicating all the data locally.

        Activity

        Hide
        David Allsopp added a comment - - edited

        Just a thought - I don't know what the overhead is with encryption enabled, but if this is a major issue, then presumably encryption could be enabled with a cypher suite that doesn't actually encrypt, thus providing just data integrity (MD5/SHA checksum) or even a 'null' encryption algorithm? I assume that this would provide authentication via the certificates, without the encryption overhead.

        Listing the supported suites using SSLContext.getServerSocketFactory().getSupportedCipherSuites() includes SSL_RSA_WITH_NULL_MD5 and SSL_RSA_WITH_NULL_SHA - which are suites used for integrity only (checksum).

        See also http://tools.ietf.org/html/rfc4785

        Show
        David Allsopp added a comment - - edited Just a thought - I don't know what the overhead is with encryption enabled, but if this is a major issue, then presumably encryption could be enabled with a cypher suite that doesn't actually encrypt , thus providing just data integrity (MD5/SHA checksum) or even a 'null' encryption algorithm? I assume that this would provide authentication via the certificates, without the encryption overhead. Listing the supported suites using SSLContext.getServerSocketFactory().getSupportedCipherSuites() includes SSL_RSA_WITH_NULL_MD5 and SSL_RSA_WITH_NULL_SHA - which are suites used for integrity only (checksum). See also http://tools.ietf.org/html/rfc4785
        Hide
        David Allsopp added a comment -

        One thing that should at least be emphasized in documentation is that the security is a bit all-or-nothing. With encryption enabled I expect the security to be solid (though I haven't investigated in detail yet - I'm unsure how easy this is to administer, or what the performance implications are).

        However, on an unencrypted cluster users may currently have a false sense of security - Thrift authentication can be enabled, but since the underlying inter-node messages are unauthenticated, I think an attacker can do what they like if they have access to the network. Iptables is therefore very important if you don't have complete control over who can access your local network. Even then, Thrift authentication can still be largely bypassed by anyone who has access to any node not blocked by iptables (i.e. any operational cluster node).

        Show
        David Allsopp added a comment - One thing that should at least be emphasized in documentation is that the security is a bit all-or-nothing. With encryption enabled I expect the security to be solid (though I haven't investigated in detail yet - I'm unsure how easy this is to administer, or what the performance implications are). However, on an unencrypted cluster users may currently have a false sense of security - Thrift authentication can be enabled, but since the underlying inter-node messages are unauthenticated, I think an attacker can do what they like if they have access to the network. Iptables is therefore very important if you don't have complete control over who can access your local network. Even then, Thrift authentication can still be largely bypassed by anyone who has access to any node not blocked by iptables (i.e. any operational cluster node).
        Hide
        Jonathan Ellis added a comment -

        Looks to me like between iptables and encryption we have this covered. Getting your ops team to use those appropriately is frankly an exercise for the reader. We don't have so many resources that we should squander them reinventing wheels that are better handled elsewhere.

        Show
        Jonathan Ellis added a comment - Looks to me like between iptables and encryption we have this covered. Getting your ops team to use those appropriately is frankly an exercise for the reader. We don't have so many resources that we should squander them reinventing wheels that are better handled elsewhere.
        Hide
        Brandon Williams added a comment -

        if you are required to use an external CA like VeriSign or GeoTrust anyone who has the ability to obtain a certificate will be able to get into your system

        There is no such requirement.

        Show
        Brandon Williams added a comment - if you are required to use an external CA like VeriSign or GeoTrust anyone who has the ability to obtain a certificate will be able to get into your system There is no such requirement.
        Hide
        Andrew Schiefelbein added a comment -

        Using encryption requires that you issue certs, so inherently you are becoming a CA when you use it.

        If your requirement for this function is that you must be your own CA then I would agree it will work, however, if you are required to use an external CA like VeriSign or GeoTrust anyone who has the ability to obtain a certificate will be able to get into your system.

        Show
        Andrew Schiefelbein added a comment - Using encryption requires that you issue certs, so inherently you are becoming a CA when you use it. If your requirement for this function is that you must be your own CA then I would agree it will work, however, if you are required to use an external CA like VeriSign or GeoTrust anyone who has the ability to obtain a certificate will be able to get into your system.
        Hide
        Brandon Williams added a comment -

        I agree that encryption is part of the solution, not the solution for this I'm sorry to say, unless you choose to become your own CA

        Using encryption requires that you issue certs, so inherently you are becoming a CA when you use it.

        it also doesn't solve the problem of we only want a select group of nodes in

        Yes, it does. Don't issue certs to nodes you don't want in the cluster, revoke them for malicious nodes.

        Show
        Brandon Williams added a comment - I agree that encryption is part of the solution, not the solution for this I'm sorry to say, unless you choose to become your own CA Using encryption requires that you issue certs, so inherently you are becoming a CA when you use it. it also doesn't solve the problem of we only want a select group of nodes in Yes, it does. Don't issue certs to nodes you don't want in the cluster, revoke them for malicious nodes.
        Hide
        Andrew Schiefelbein added a comment -

        I do agree that is a solution that would work for many but I will have to echo my previous statement about being unable to modify the host system or network, it is far easier for myself, and I'm quite sure a multitude of others, to edit a properties file or a cassandra column family than it is to get root level authority to change the iptables of a system.

        Show
        Andrew Schiefelbein added a comment - I do agree that is a solution that would work for many but I will have to echo my previous statement about being unable to modify the host system or network, it is far easier for myself, and I'm quite sure a multitude of others, to edit a properties file or a cassandra column family than it is to get root level authority to change the iptables of a system.
        Hide
        Rick Branson added a comment -

        It sounds like the interest is in an IP whitelist, so I'll echo the previous statement and agree that using iptables is the best route to take.

        Show
        Rick Branson added a comment - It sounds like the interest is in an IP whitelist, so I'll echo the previous statement and agree that using iptables is the best route to take.
        Hide
        David Allsopp added a comment - - edited

        D'oh, I hadn't realized encryption was implemented now! Spent too much time on old versions...

        Verifying IP addresses might still be useful as per the sub-issue, but it's a minor point if proper encryption is in place.

        Show
        David Allsopp added a comment - - edited D'oh, I hadn't realized encryption was implemented now! Spent too much time on old versions... Verifying IP addresses might still be useful as per the sub-issue, but it's a minor point if proper encryption is in place.
        Hide
        Andrew Schiefelbein added a comment - - edited

        Good point about storing the settings in the Cassandra storage itself, though it looks from https://issues.apache.org/jira/browse/CASSANDRA-3319 as though it needs to be a non-system keyspace . It feels wrong for it to be a 'normal' keyspace though, as it would need to have a predetermined name and schema in order for various parts of the server code to use if when authenticating, and it would have the same visibility and access as normal data keyspaces (surely it should require greater privileges, as for schema modifications and other 'dangerous' operations?).

        If you go according to the OpenLDAP model there is a separate LDAP database that contains the settings that is used by the underlying system, it has a separate login than the normal database that it's serving and never the two shall meet. To translate that to Cassandra there would have to be a separate keyspace or maybe in this case possibly a "database" that is serving the cluster wide options. I haven't really thought through the different permutations on this, but I would hope it is achievable.

        It seems to me, if you really want to do this properly, you should enable the existing encryption options.

        I agree that encryption is part of the solution, not the solution for this I'm sorry to say, unless you choose to become your own CA or have something that checks an embedded thing in, oh say, the OU level of the certificate (encrypted has of course, otherwise it's clear text), it is fairly easy to get a signed certificate that will pass through the normal SSL handshake without question, it also doesn't solve the problem of we only want a select group of nodes in. If there is something like Apache HTTPD's SSLRequire (http://httpd.apache.org/docs/2.0/mod/mod_ssl.html#sslrequire) that allows for some form of regular expression that checks a number of things that may actually be pretty close to what I was suggesting, but you would still need to maintain the list of stuff you want checked cluster wide. Only those nodes you want to talk to should be allowed in, all others should be told to go away or you will taunt them a second time.

        Show
        Andrew Schiefelbein added a comment - - edited Good point about storing the settings in the Cassandra storage itself, though it looks from https://issues.apache.org/jira/browse/CASSANDRA-3319 as though it needs to be a non-system keyspace . It feels wrong for it to be a 'normal' keyspace though, as it would need to have a predetermined name and schema in order for various parts of the server code to use if when authenticating, and it would have the same visibility and access as normal data keyspaces (surely it should require greater privileges, as for schema modifications and other 'dangerous' operations?). If you go according to the OpenLDAP model there is a separate LDAP database that contains the settings that is used by the underlying system, it has a separate login than the normal database that it's serving and never the two shall meet. To translate that to Cassandra there would have to be a separate keyspace or maybe in this case possibly a "database" that is serving the cluster wide options. I haven't really thought through the different permutations on this, but I would hope it is achievable. It seems to me, if you really want to do this properly, you should enable the existing encryption options. I agree that encryption is part of the solution, not the solution for this I'm sorry to say, unless you choose to become your own CA or have something that checks an embedded thing in, oh say, the OU level of the certificate (encrypted has of course, otherwise it's clear text), it is fairly easy to get a signed certificate that will pass through the normal SSL handshake without question, it also doesn't solve the problem of we only want a select group of nodes in. If there is something like Apache HTTPD's SSLRequire ( http://httpd.apache.org/docs/2.0/mod/mod_ssl.html#sslrequire ) that allows for some form of regular expression that checks a number of things that may actually be pretty close to what I was suggesting, but you would still need to maintain the list of stuff you want checked cluster wide. Only those nodes you want to talk to should be allowed in, all others should be told to go away or you will taunt them a second time.
        Hide
        Brandon Williams added a comment -

        I can't see how it prevents attacks from trusted nodes

        If the node is in fact trusted then you're screwed. Add as many layers as you like, but the definition of trusted won't change and you'll have the same core problem: something trusted is doing something malicious.

        how would the use of encryption work in a multi-tenant environment?

        I don't understand, a cluster is a cluster regardless of the amount of tenants. It sounds like you're describing something where multiple tenants would each contribute nodes to the cluster, that is not supported.

        The standard practice of rolling encryption keys might be difficult as well

        If the machine is trusted and compromised and wants to be malicious, it's game over, rolling keys or not. I don't see any value in adding such complexity given this.

        If a key becomes compromised then a receiving node will still honor a sender's request because it doesn't attempt to do any verification that the sender has the authority to performing the action it is requesting be executed.

        Then the CA should issue a revocation for that key. SSL/TLS has these problems solved.

        Show
        Brandon Williams added a comment - I can't see how it prevents attacks from trusted nodes If the node is in fact trusted then you're screwed. Add as many layers as you like, but the definition of trusted won't change and you'll have the same core problem: something trusted is doing something malicious . how would the use of encryption work in a multi-tenant environment? I don't understand, a cluster is a cluster regardless of the amount of tenants. It sounds like you're describing something where multiple tenants would each contribute nodes to the cluster, that is not supported. The standard practice of rolling encryption keys might be difficult as well If the machine is trusted and compromised and wants to be malicious, it's game over, rolling keys or not. I don't see any value in adding such complexity given this. If a key becomes compromised then a receiving node will still honor a sender's request because it doesn't attempt to do any verification that the sender has the authority to performing the action it is requesting be executed. Then the CA should issue a revocation for that key. SSL/TLS has these problems solved.
        Hide
        Richard Lowe added a comment -

        I'd forgotten about the encryption options. Enabling encryption would certainly prevent against rogue nodes entering the cluster (unless they had a valid key) but I can't see how it prevents attacks from trusted nodes, i.e. through a malicious/erronous/compromised app or user action. Such nodes would still be able to request an action that could (or should) require specific privileges.

        In our particular use case a drop of a keyspace without the authority to do so would be very bad, whilst it sounds like the original reporter of this issue is concerned about the potential for leaking data through unauthorized reads. I'd be surprised if there weren't other applications for which deleting the database wouldn't cause a problem; at the very least it means downtime to restore from the last set of snapshots.

        Also, how would the use of encryption work in a multi-tenant environment? Would each tenant have their own set of encryption keys? Does this sort of segregation of the cluster cause problems in terms of gossip and thrift messaging?

        The standard practice of rolling encryption keys might be difficult as well, requiring each node to be stopped and restarted for updates to its keystore to take effect. This doesn't satisfy the desire for the provision of security to be dynamically applied, as Andrew Scheifelbein described previously.

        If a key becomes compromised then a receiving node will still honor a sender's request because it doesn't attempt to do any verification that the sender has the authority to performing the action it is requesting be executed.

        Encryption gets us quite a lot of the way, but it doesn't get us the whole way there in terms of ensuring a node is only able to perform the actions it is authorized to do so upon the cluster.

        Show
        Richard Lowe added a comment - I'd forgotten about the encryption options. Enabling encryption would certainly prevent against rogue nodes entering the cluster (unless they had a valid key) but I can't see how it prevents attacks from trusted nodes, i.e. through a malicious/erronous/compromised app or user action. Such nodes would still be able to request an action that could (or should) require specific privileges. In our particular use case a drop of a keyspace without the authority to do so would be very bad, whilst it sounds like the original reporter of this issue is concerned about the potential for leaking data through unauthorized reads. I'd be surprised if there weren't other applications for which deleting the database wouldn't cause a problem; at the very least it means downtime to restore from the last set of snapshots. Also, how would the use of encryption work in a multi-tenant environment? Would each tenant have their own set of encryption keys? Does this sort of segregation of the cluster cause problems in terms of gossip and thrift messaging? The standard practice of rolling encryption keys might be difficult as well, requiring each node to be stopped and restarted for updates to its keystore to take effect. This doesn't satisfy the desire for the provision of security to be dynamically applied, as Andrew Scheifelbein described previously. If a key becomes compromised then a receiving node will still honor a sender's request because it doesn't attempt to do any verification that the sender has the authority to performing the action it is requesting be executed. Encryption gets us quite a lot of the way, but it doesn't get us the whole way there in terms of ensuring a node is only able to perform the actions it is authorized to do so upon the cluster.
        Hide
        Brandon Williams added a comment -

        It seems to me, if you really want to do this properly, you should enable the existing encryption options.

        Show
        Brandon Williams added a comment - It seems to me, if you really want to do this properly, you should enable the existing encryption options.
        Hide
        David Allsopp added a comment - - edited

        One additional issue is that currently we deserialize the "from" IP address from the incoming message header, using Header.deserialize() and CompactEndpointSerializationHelper.deserialize() i.e. we trust the sender to supply a true IP address.

        Might be worth a sub-issue to change this, i.e. don't store the IP address in the message Header at all (saving a small amount of space) and set the 'true' sender IP upon receipt of the message, in org.apache.cassandra.net.IncomingTcpConnection, using socket.getInetAddress() ?

        Show
        David Allsopp added a comment - - edited One additional issue is that currently we deserialize the "from" IP address from the incoming message header, using Header.deserialize() and CompactEndpointSerializationHelper.deserialize() i.e. we trust the sender to supply a true IP address . Might be worth a sub-issue to change this, i.e. don't store the IP address in the message Header at all (saving a small amount of space) and set the 'true' sender IP upon receipt of the message, in org.apache.cassandra.net.IncomingTcpConnection, using socket.getInetAddress() ?
        Hide
        David Allsopp added a comment - - edited

        Using a custom PROTOCOL_MAGIC would be a very easy way to improve matters substantially. Would be better if it was at least a long, not an int though, otherwise you could brute force it, given a few weeks.

        However, the weakness of this is that the shared secret (PROTOCOL_MAGIC) is constant, and is exposed in every message, so an attacker only needs to see one message to defeat the system.

        A hash of the source IP and a shared secret would be better, as then the secret isn't transmitted in the clear, and the IP can be validated as part of the authentication.

        In either case, another weakness is that the secret can't be easily changed without cluster-wide reconfiguration and restart.

        With a bit more work, we could combine the above with a list of allowed IPs, plus a per-node secret, so that we only need reconfiguration when adding or removing nodes, and can hopefully reconfigure on-the-fly. So each message would contain a hash of the source node's IP and secret. This would only need to be computed once by the source node and once by each other node, then cached.

        Show
        David Allsopp added a comment - - edited Using a custom PROTOCOL_MAGIC would be a very easy way to improve matters substantially. Would be better if it was at least a long , not an int though, otherwise you could brute force it, given a few weeks. However, the weakness of this is that the shared secret (PROTOCOL_MAGIC) is constant, and is exposed in every message, so an attacker only needs to see one message to defeat the system. A hash of the source IP and a shared secret would be better, as then the secret isn't transmitted in the clear, and the IP can be validated as part of the authentication. In either case, another weakness is that the secret can't be easily changed without cluster-wide reconfiguration and restart. With a bit more work, we could combine the above with a list of allowed IPs, plus a per-node secret, so that we only need reconfiguration when adding or removing nodes, and can hopefully reconfigure on-the-fly. So each message would contain a hash of the source node's IP and secret. This would only need to be computed once by the source node and once by each other node, then cached.
        Hide
        Richard Lowe added a comment -

        I took a look at this too and also identified o.a.c.net.MessagingService as the best place to start adding receiver-side security. At the moment the authentication provided by o.a.c.auth.IAuthority and o.a.c.auth.IAuthenticator is client-side only, so no help in preventing nodes from processing requests from rogue or malicious nodes that have their authentication turned off (the default).

        There are two approaches I identified for preventing unauthorised nodes from interacting with the cluster.

        1. The MessagingService defines a PROTOCOL_MAGIC value that it uses to determine if the node sending an incoming message is "sane". By changing this value from the default value it would effectively mean that only nodes running your own modified version of Cassandra would be able to communicate with each other, as all other nodes (including those using an official release) would treat your nodes as "insane" and refuse to communicate. This is a brutal way to achieve the goal, but it may be sufficient for some.
        2. Add some checking of messages that are received before the action they request is carried out. At the moment the receive() method in MessagingService does nothing to verify the authenticity or authority of an incoming message before queuing it for processing. However, the o.a.c.net.Message that is passed to the receive() method has a from field, which as David Allsopp points out contains the IP address of the receiving node. If the IP address is sufficient to ascertain the identity of the sender (for your use case) then this could form the basis of some validation, e.g. checking that the IP is in a list of allowed IPs (or range of IPs), either stored in a configuration file or, as has been suggested, in a cluster-wide shared column family (although I'm not quite sure how that would work in practice).

        If IP address alone is not enough to identify a node (which is highly likely), then I think there is no choice but to extend the Message class to include extra identity and credentials information.

        I'm not sure how feasible this is as I'm not familiar enough with Cassandra and its underlying protocols: if secured nodes were to use an extension of Message instead of Message itself then it's likely that unsecure nodes would not be able to process messages sent by secure nodes – but then perhaps that's the point.

        When it comes to validating identity and credentials, it seems sensible to re-use existing PKI or similar mechanisms, through PAM integration to take advantage of system policies and AD or LDAP. The strategy used should be configurable and pluggable, just as the existing client-side security strategy is, and include the ability to turn off sender verification to allow the current status quo to continue in environments where that makes sense.

        Unfortunately I don't have time to implement any of this right now, but really wish I did. This is an important aspect for us though so I may be able to have a go soon, if someone else doesn't beat me to it.

        Show
        Richard Lowe added a comment - I took a look at this too and also identified o.a.c.net.MessagingService as the best place to start adding receiver-side security. At the moment the authentication provided by o.a.c.auth.IAuthority and o.a.c.auth.IAuthenticator is client-side only, so no help in preventing nodes from processing requests from rogue or malicious nodes that have their authentication turned off (the default). There are two approaches I identified for preventing unauthorised nodes from interacting with the cluster. The MessagingService defines a PROTOCOL_MAGIC value that it uses to determine if the node sending an incoming message is "sane". By changing this value from the default value it would effectively mean that only nodes running your own modified version of Cassandra would be able to communicate with each other, as all other nodes (including those using an official release) would treat your nodes as "insane" and refuse to communicate. This is a brutal way to achieve the goal, but it may be sufficient for some. Add some checking of messages that are received before the action they request is carried out. At the moment the receive() method in MessagingService does nothing to verify the authenticity or authority of an incoming message before queuing it for processing. However, the o.a.c.net.Message that is passed to the receive() method has a from field, which as David Allsopp points out contains the IP address of the receiving node. If the IP address is sufficient to ascertain the identity of the sender (for your use case) then this could form the basis of some validation, e.g. checking that the IP is in a list of allowed IPs (or range of IPs), either stored in a configuration file or, as has been suggested, in a cluster-wide shared column family (although I'm not quite sure how that would work in practice). If IP address alone is not enough to identify a node (which is highly likely), then I think there is no choice but to extend the Message class to include extra identity and credentials information. I'm not sure how feasible this is as I'm not familiar enough with Cassandra and its underlying protocols: if secured nodes were to use an extension of Message instead of Message itself then it's likely that unsecure nodes would not be able to process messages sent by secure nodes – but then perhaps that's the point. When it comes to validating identity and credentials, it seems sensible to re-use existing PKI or similar mechanisms, through PAM integration to take advantage of system policies and AD or LDAP. The strategy used should be configurable and pluggable, just as the existing client-side security strategy is, and include the ability to turn off sender verification to allow the current status quo to continue in environments where that makes sense. Unfortunately I don't have time to implement any of this right now, but really wish I did. This is an important aspect for us though so I may be able to have a go soon, if someone else doesn't beat me to it.
        Hide
        David Allsopp added a comment - - edited

        From a quick scan through the code, it looks as though we'd need MessagingService (and possibly JMX) to check the list of IPs or a supplied token, in addition to the existing Thrift authentication.

        Currently it appears that (in the absence of a firewall) there is no security against reading data from a cluster - the Thrift interface requires authentication, but the MessagingService doesn't, so I think you can just send Read messages etc to pull data out of a cluster. Is that right, or is there something I've missed?

        I think we need to be precise about what threat we are trying to defend against, so we can weight up whether it's worth the effort. For example, does our hypothetical attacker have root access to a host on the LAN? If so, then presumably they may be able to spoof IP addresses, completely bypassing any IP checks? What happens if they simply set their IP to the same as that of one of the genuine nodes? I assume that you can defend against this with a suitably configured switch/router (with Sticky ARP / Dynamic ARP Inspection), but our scenario here seems to be that we don't have control over this sort of detail (which makes any kind of security very difficult). If our attacker has the same privileges as us, then IP-based checks are probably OK.

        Note also that none of the defences discussed so far would defend against denial-of-service attacks from an insider on the network (by trying to overload or crash nodes in the cluster).

        Physical attacks require physical defences - if untrusted people have access to your hardware then it's game over.

        Show
        David Allsopp added a comment - - edited From a quick scan through the code, it looks as though we'd need MessagingService (and possibly JMX) to check the list of IPs or a supplied token, in addition to the existing Thrift authentication. Currently it appears that (in the absence of a firewall) there is no security against reading data from a cluster - the Thrift interface requires authentication, but the MessagingService doesn't, so I think you can just send Read messages etc to pull data out of a cluster. Is that right, or is there something I've missed? I think we need to be precise about what threat we are trying to defend against, so we can weight up whether it's worth the effort. For example, does our hypothetical attacker have root access to a host on the LAN? If so, then presumably they may be able to spoof IP addresses, completely bypassing any IP checks? What happens if they simply set their IP to the same as that of one of the genuine nodes? I assume that you can defend against this with a suitably configured switch/router (with Sticky ARP / Dynamic ARP Inspection), but our scenario here seems to be that we don't have control over this sort of detail (which makes any kind of security very difficult). If our attacker has the same privileges as us, then IP-based checks are probably OK. Note also that none of the defences discussed so far would defend against denial-of-service attacks from an insider on the network (by trying to overload or crash nodes in the cluster). Physical attacks require physical defences - if untrusted people have access to your hardware then it's game over.
        Hide
        David Allsopp added a comment -

        Good point about storing the settings in the Cassandra storage itself, though it looks from https://issues.apache.org/jira/browse/CASSANDRA-3319 as though it needs to be a non-system keyspace . It feels wrong for it to be a 'normal' keyspace though, as it would need to have a predetermined name and schema in order for various parts of the server code to use if when authenticating, and it would have the same visibility and access as normal data keyspaces (surely it should require greater privileges, as for schema modifications and other 'dangerous' operations?).

        Show
        David Allsopp added a comment - Good point about storing the settings in the Cassandra storage itself, though it looks from https://issues.apache.org/jira/browse/CASSANDRA-3319 as though it needs to be a non-system keyspace . It feels wrong for it to be a 'normal' keyspace though, as it would need to have a predetermined name and schema in order for various parts of the server code to use if when authenticating, and it would have the same visibility and access as normal data keyspaces (surely it should require greater privileges, as for schema modifications and other 'dangerous' operations?).
        Hide
        Andrew Schiefelbein added a comment -

        I believe Bob Blakley once wrote that trust is for suckers. I'm not worried about physical attacks against a single box, you can never be 100% secure, because if there is a way there is a will, however, the current security model allows enough of a hole for a sophisticated individual with enough knowledge and malicious intent to quite easy fire up a node and replicate off data without so much as a slap on the wrist. I believe that point is taken on this thread so I will stop hammering it home. That said, I will award bonus points and buy you a 6 pack of whatever tasty beverage you prefer if you could enable this cluster wide, and have these settings dynamic in the database itself that you can tweak while it's running. Bringing nodes up and down to add / remove other nodes is no fun, as it is also no fun when you have to modify users and access properties. And though I was invited to help fix this here:
        https://issues.apache.org/jira/browse/CASSANDRA-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
        I haven't been able to win the lottery to get enough free time to do it, hence the offer of the 6 pack.

        Show
        Andrew Schiefelbein added a comment - I believe Bob Blakley once wrote that trust is for suckers. I'm not worried about physical attacks against a single box, you can never be 100% secure, because if there is a way there is a will, however, the current security model allows enough of a hole for a sophisticated individual with enough knowledge and malicious intent to quite easy fire up a node and replicate off data without so much as a slap on the wrist. I believe that point is taken on this thread so I will stop hammering it home. That said, I will award bonus points and buy you a 6 pack of whatever tasty beverage you prefer if you could enable this cluster wide, and have these settings dynamic in the database itself that you can tweak while it's running. Bringing nodes up and down to add / remove other nodes is no fun, as it is also no fun when you have to modify users and access properties. And though I was invited to help fix this here: https://issues.apache.org/jira/browse/CASSANDRA-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel I haven't been able to win the lottery to get enough free time to do it, hence the offer of the 6 pack.
        Hide
        David Allsopp added a comment -

        @Mark - There's a convenience/security trade-off there. Giving each node a security token is easier and more flexible than configuring every node with a list of all other nodes (it's a bit of a pain to have to update all existing nodes each time you want to add a node). However, the security in this scenario is to deal with a semi-insider attack. Getting hold of the token might be easier for the attacker than having to somehow modify the access list on all existing nodes.

        However, what stops an attacker from modifying the access list on all existing nodes? Only the login/password for those nodes. So in either case the weakest link is the protection of a single password/token (maybe multiple passwords if you have a different login for each node, but is that commonplace??).

        Care would also be needed to make sure the shared secret isn't exposed over the network, either as plaintext, or via replay of messages from existing nodes.

        So whilst I think your proposal is appealing from the operational efficiency point of view, it is probably harder to implement securely.

        Trying to protect against an insider who might even have physical access to your servers is a very tough game...

        Show
        David Allsopp added a comment - @Mark - There's a convenience/security trade-off there. Giving each node a security token is easier and more flexible than configuring every node with a list of all other nodes (it's a bit of a pain to have to update all existing nodes each time you want to add a node). However, the security in this scenario is to deal with a semi-insider attack. Getting hold of the token might be easier for the attacker than having to somehow modify the access list on all existing nodes. However, what stops an attacker from modifying the access list on all existing nodes? Only the login/password for those nodes. So in either case the weakest link is the protection of a single password/token (maybe multiple passwords if you have a different login for each node, but is that commonplace??). Care would also be needed to make sure the shared secret isn't exposed over the network, either as plaintext, or via replay of messages from existing nodes. So whilst I think your proposal is appealing from the operational efficiency point of view, it is probably harder to implement securely. Trying to protect against an insider who might even have physical access to your servers is a very tough game...
        Hide
        Mark Allsopp added a comment - - edited

        New to using cassandra at the moment, but this does cover an issue we are seeing and I think the original comment seems valid for a number of cases.

        I may be extending this a little too far, but is there a possibility of perhaps going a little bit further than just a host list and extending it to a basic remote node authentication. By default not trusting any new node that joins and asks it to prove that it's a valid node.

        Perhaps something as simple as just using a simple secret "password token" such as a random character string that identifies the node as a member of a cluster (and possibly even keyspace) and refusing access if it's not present would start to kill two birds with a single stone.

        Initially it could reply on an user keeping a properties files with the cluster to token and keyspace to token mappings on each box. Kind of extending simple auth to being simple remote auth.

        Obviously there are still holes in what I'm suggesting from a security point of view, but something that requires a new node to provide some degree of likelihood that it really is who is says it is would be great.

        Show
        Mark Allsopp added a comment - - edited New to using cassandra at the moment, but this does cover an issue we are seeing and I think the original comment seems valid for a number of cases. I may be extending this a little too far, but is there a possibility of perhaps going a little bit further than just a host list and extending it to a basic remote node authentication. By default not trusting any new node that joins and asks it to prove that it's a valid node. Perhaps something as simple as just using a simple secret "password token" such as a random character string that identifies the node as a member of a cluster (and possibly even keyspace) and refusing access if it's not present would start to kill two birds with a single stone. Initially it could reply on an user keeping a properties files with the cluster to token and keyspace to token mappings on each box. Kind of extending simple auth to being simple remote auth. Obviously there are still holes in what I'm suggesting from a security point of view, but something that requires a new node to provide some degree of likelihood that it really is who is says it is would be great.
        Hide
        David Allsopp added a comment - - edited

        OK, I understand now!

        Just a few things to clarify my understanding:

        As far as I can see from the docs online, the Cassandra authentication is configured on a per-node basis, and is used to control Thrift/CQL client access, but not inter-node messages (hence it is possible to add a rogue node as you describe above)?

        Once this new node starts accepting data, its SSTables will presumably be accessible to the attacker, and the subset of data held on that node could then be extracted without any userid and password? And, given plenty of time, and assuming nobody is paying attention to nodetool etc, one could move this node around the ring to get a sampling of the data from around the keyspace.

        Finally, if the rogue node is queried via Thrift/CQL, presumably it won't be configured to require any userid/password. When it passes on queries for rows held on other nodes, will those nodes require credentials as if the rogue node were a client contacting them directly? Or would these inter-node queries bypass the authentication? If the former, how does that work, since the authentication is configured (I think) separately on each client, so the credentials coudl be different? If the latter, then the rogue node bypasses all authentication.

        Show
        David Allsopp added a comment - - edited OK, I understand now! Just a few things to clarify my understanding: As far as I can see from the docs online, the Cassandra authentication is configured on a per-node basis, and is used to control Thrift/CQL client access, but not inter-node messages (hence it is possible to add a rogue node as you describe above)? Once this new node starts accepting data, its SSTables will presumably be accessible to the attacker, and the subset of data held on that node could then be extracted without any userid and password? And, given plenty of time, and assuming nobody is paying attention to nodetool etc, one could move this node around the ring to get a sampling of the data from around the keyspace. Finally, if the rogue node is queried via Thrift/CQL, presumably it won't be configured to require any userid/password. When it passes on queries for rows held on other nodes, will those nodes require credentials as if the rogue node were a client contacting them directly? Or would these inter-node queries bypass the authentication? If the former, how does that work, since the authentication is configured (I think) separately on each client, so the credentials coudl be different? If the latter, then the rogue node bypasses all authentication.
        Hide
        Andrew Schiefelbein added a comment -

        Though I agree that you could fix this particular problem by walling off the systems I don't believe that is a very workable solution for larger environments where all you have is the ability to log in as a named user to a system in the data center, and you do not have root access and if you wish to setup firewalls the best that can said to you is good luck with that. The thing that I was pushing for in my original post was to have the ability to control access without modification to a host system, or host network, and to be able to do this as an enfeebled user and not a super one. I agree with you that if this was available you would do one or the other, doing both would cause no end of problems, but to have the ability to control this as a normal user makes life easier for those of us who have to play nicely in a very small sandbox.

        Show
        Andrew Schiefelbein added a comment - Though I agree that you could fix this particular problem by walling off the systems I don't believe that is a very workable solution for larger environments where all you have is the ability to log in as a named user to a system in the data center, and you do not have root access and if you wish to setup firewalls the best that can said to you is good luck with that. The thing that I was pushing for in my original post was to have the ability to control access without modification to a host system, or host network, and to be able to do this as an enfeebled user and not a super one. I agree with you that if this was available you would do one or the other, doing both would cause no end of problems, but to have the ability to control this as a normal user makes life easier for those of us who have to play nicely in a very small sandbox.
        Hide
        David Allsopp added a comment -

        Can this not be done simply via iptables firewall settings, rolled out across the cluster machines?

        (It's not clear to me why a Cassandra configuration file would be any less fallible than a firewall configuration file - and if you have both, then you have to remember to update both of them...)

        Show
        David Allsopp added a comment - Can this not be done simply via iptables firewall settings, rolled out across the cluster machines? (It's not clear to me why a Cassandra configuration file would be any less fallible than a firewall configuration file - and if you have both, then you have to remember to update both of them...)

          People

          • Assignee:
            Unassigned
            Reporter:
            Andrew Schiefelbein
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development