Solr
  1. Solr
  2. SOLR-4580

Support for protecting content in ZK

    Details

      Description

      We want to protect content in zookeeper.

      In order to run a CloudSolrServer in "client-space" you will have to open for access to zookeeper from client-space.
      If you do not trust persons or systems in client-space you want to protect zookeeper against evilness from client-space - e.g.

      • Changing configuration
      • Trying to mess up system by manipulating clusterstate
      • Add a delete-collection job to be carried out by the Overseer
      • etc

      Even if you do not open for zookeeper access to someone outside your "secure zone" you might want to protect zookeeper content from being manipulated by e.g.

      • Malware that found its way into secure zone
      • Other systems also using zookeeper
      • etc.
      1. SOLR-4580_branch_4x_r1482255.patch
        61 kB
        Per Steffensen
      2. SOLR-4580.patch
        73 kB
        Mark Miller
      3. SOLR-4580.patch
        82 kB
        Mark Miller
      4. SOLR-4580.patch
        87 kB
        Mark Miller

        Activity

        Hide
        Per Steffensen added a comment -

        Early implemention thoughts

        • You have to actively do something to use protective ACLs - default will still be ZooDefs.Ids.OPEN_ACL_UNSAFE allover
        • Separate ACL-provider/factory component (for nice "separation of concern") with a default implementation (in time, if someone needs it, we can expose the ACL-provider/factory-implementation to be replaceable/pluggable from "the outside" - e.g. though solrconfig.xml). Default implementation could be something like this:
          • Operate with two sets of "digest"-credentials (username/password-based) to be used for ZK access - internal-user and client-user
            • internal-user: Will be allowed to do anything in the solr-space in ZK
            • client-user: Read-only. Either allowed it to read-only anything in the solr-space in ZK, or tighten it a little and only allow it to read-only "whatever is necessary for a CloudSolrServer to operate" (e.g. not access to read Overseer stuff and configuration stuff)
        • Solr-nodes will need to know about both internal-user and client-user credentials
          • internal-user to be used for its own connetions to ZK, and to create "everything"-ACLs for this internal-user on every znodes it creates
          • client-user to create "read-only"-ACLs for this client-user on every (or selected) znodes it creates
        • CloudSolrServers used outside Solr-nodes will need client-user only, to be used on its ZK-connections, to be able to read necessary stuff
        • ZkCLI needs to know about internal-user credentials to be used for its own connections to ZK, and to create "everything"-ACLs for this internal-user on every znodes it creates
        Show
        Per Steffensen added a comment - Early implemention thoughts You have to actively do something to use protective ACLs - default will still be ZooDefs.Ids.OPEN_ACL_UNSAFE allover Separate ACL-provider/factory component (for nice "separation of concern") with a default implementation (in time, if someone needs it, we can expose the ACL-provider/factory-implementation to be replaceable/pluggable from "the outside" - e.g. though solrconfig.xml). Default implementation could be something like this: Operate with two sets of "digest"-credentials (username/password-based) to be used for ZK access - internal-user and client-user internal-user: Will be allowed to do anything in the solr-space in ZK client-user: Read-only. Either allowed it to read-only anything in the solr-space in ZK, or tighten it a little and only allow it to read-only "whatever is necessary for a CloudSolrServer to operate" (e.g. not access to read Overseer stuff and configuration stuff) Solr-nodes will need to know about both internal-user and client-user credentials internal-user to be used for its own connetions to ZK, and to create "everything"-ACLs for this internal-user on every znodes it creates client-user to create "read-only"-ACLs for this client-user on every (or selected) znodes it creates CloudSolrServers used outside Solr-nodes will need client-user only, to be used on its ZK-connections, to be able to read necessary stuff ZkCLI needs to know about internal-user credentials to be used for its own connections to ZK, and to create "everything"-ACLs for this internal-user on every znodes it creates
        Hide
        Per Steffensen added a comment -

        Make a short description on Solr Security Wiki page about how to use

        Show
        Per Steffensen added a comment - Make a short description on Solr Security Wiki page about how to use
        Hide
        Per Steffensen added a comment -

        Patch fitting branch_4x revision 1482255. A little bit of documentation on why the solution ended up as it did, how it works and how to use it will follow shortly - as a comment here or on Wiki.

        Show
        Per Steffensen added a comment - Patch fitting branch_4x revision 1482255. A little bit of documentation on why the solution ended up as it did, how it works and how to use it will follow shortly - as a comment here or on Wiki.
        Show
        Per Steffensen added a comment - Documentation: https://wiki.apache.org/solr/Per%20Steffensen/ZooKeeper%20protecting%20content
        Hide
        Mark Miller added a comment -

        I'll review this issue.

        Show
        Mark Miller added a comment - I'll review this issue.
        Hide
        Mark Miller added a comment -

        Looks good, patch still applies and applies to trunk.

        I'll commit this soon unless someone has further comments.

        Show
        Mark Miller added a comment - Looks good, patch still applies and applies to trunk. I'll commit this soon unless someone has further comments.
        Hide
        Mark Miller added a comment -

        Patch attached:
        I've made some very small tweaks/fixes and added a new dev start scripts that runs a cloud cluster against a secured zk.

        I tried to access the zk info without credentials from another zk tool with this setup - I could only see the root nodes, none of their children. I could not read the contents of the nodes. I could not delete nodes that had children (I could not see the children but got a relevant error when trying to delete nodes with children).

        Oddly, it seems I could delete nodes with no children - for example cluster.json and alias.json.

        Show
        Mark Miller added a comment - Patch attached: I've made some very small tweaks/fixes and added a new dev start scripts that runs a cloud cluster against a secured zk. I tried to access the zk info without credentials from another zk tool with this setup - I could only see the root nodes, none of their children. I could not read the contents of the nodes. I could not delete nodes that had children (I could not see the children but got a relevant error when trying to delete nodes with children). Oddly, it seems I could delete nodes with no children - for example cluster.json and alias.json.
        Hide
        Mark Miller added a comment -

        Oddly, it seems I could delete nodes with no children - for example cluster.json and alias.json.

        I'm guessing this is perhaps because the root node has no permissions - and that's what lets me see the shallow listing to begin with - and what let's me do the delete - though that's still mildly surprising. In which case you want to protect the root node somehow or use a chroot.

        Show
        Mark Miller added a comment - Oddly, it seems I could delete nodes with no children - for example cluster.json and alias.json. I'm guessing this is perhaps because the root node has no permissions - and that's what lets me see the shallow listing to begin with - and what let's me do the delete - though that's still mildly surprising. In which case you want to protect the root node somehow or use a chroot.
        Hide
        Mark Miller added a comment -

        I've updated this to trunk. Patch coming after I do some testing again.

        Show
        Mark Miller added a comment - I've updated this to trunk. Patch coming after I do some testing again.
        Hide
        Mark Miller added a comment -

        Here is a patch updated to trunk.

        If you try and use zk security without a chroot, it fails to start.

        I still have to test using custom zk security impls and review the way custom impls classloading is done.

        Show
        Mark Miller added a comment - Here is a patch updated to trunk. If you try and use zk security without a chroot, it fails to start. I still have to test using custom zk security impls and review the way custom impls classloading is done.
        Hide
        Mark Miller added a comment -

        Oh yeah, I also worked the cloud-dev script to test this manually into the exisiting standard scripts by using the new JAVA_OPTS support instead of a new one off script. This is also how you can make cloud-dev scripts work with HDFS and makes things much easier to test and maintain.

        Show
        Mark Miller added a comment - Oh yeah, I also worked the cloud-dev script to test this manually into the exisiting standard scripts by using the new JAVA_OPTS support instead of a new one off script. This is also how you can make cloud-dev scripts work with HDFS and makes things much easier to test and maintain.
        Hide
        Per Steffensen added a comment -

        Happy to see some activity on this one. It might not be a feature that many will use - it is for the extra paranoid, but it would be nice to support it in Solr. We have been using it for a long time and it works like a charm. We use it with a chroot (/solr) so there might be a few things that need a twist if you want your solr stuff directly in the root of ZK. We dont want that because we have numerous other non-Solr stuff in ZK for which we use separate sets of username/password, so that the different applications do not accidently mess up or see each others ZK content. I would recommend in general that Solr users do not keep the Solr content directly in ZK root.

        The support for custom impl is important for us. We are very paranoid and have an implementation where admins punch in password when they start Solr. The passwords are streamed into Solr and split secs after Solr has been started the passwords exist nowhere but in JVM memory. Not on command-line, not in config-files or any other places.

        In lack of better ZK clients (let me know if you know any good ZK clients) I use http://www.massedynamic.org/mediawiki/index.php?title=Eclipse_Plug-in_for_ZooKeeper (update site: http://www.massedynamic.org/eclipse/updates/) Eclipse plugin for messing around in ZK. It is not perfect (can be a little slow at times), but it works ok. It might be of value for you to use that as well.

        Let me know if you need any help.

        Show
        Per Steffensen added a comment - Happy to see some activity on this one. It might not be a feature that many will use - it is for the extra paranoid, but it would be nice to support it in Solr. We have been using it for a long time and it works like a charm. We use it with a chroot (/solr) so there might be a few things that need a twist if you want your solr stuff directly in the root of ZK. We dont want that because we have numerous other non-Solr stuff in ZK for which we use separate sets of username/password, so that the different applications do not accidently mess up or see each others ZK content. I would recommend in general that Solr users do not keep the Solr content directly in ZK root. The support for custom impl is important for us. We are very paranoid and have an implementation where admins punch in password when they start Solr. The passwords are streamed into Solr and split secs after Solr has been started the passwords exist nowhere but in JVM memory. Not on command-line, not in config-files or any other places. In lack of better ZK clients (let me know if you know any good ZK clients) I use http://www.massedynamic.org/mediawiki/index.php?title=Eclipse_Plug-in_for_ZooKeeper (update site: http://www.massedynamic.org/eclipse/updates/ ) Eclipse plugin for messing around in ZK. It is not perfect (can be a little slow at times), but it works ok. It might be of value for you to use that as well. Let me know if you need any help.
        Hide
        Shawn Heisey added a comment -

        I would recommend in general that Solr users do not keep the Solr content directly in ZK root.

        Strong +1 from me on this. I think a chroot should be in all the documentation and every SolrCloud example maintained by the project. I can understand if that's not the way we go, but I really think we should.

        Show
        Shawn Heisey added a comment - I would recommend in general that Solr users do not keep the Solr content directly in ZK root. Strong +1 from me on this. I think a chroot should be in all the documentation and every SolrCloud example maintained by the project. I can understand if that's not the way we go, but I really think we should.
        Hide
        Mark Miller added a comment -

        If you try and use zk security without a chroot, it fails to start.

        I realize I may have been unclear here. It fails to start because I added code to make it so - if you attempt to configure zk/solr security and are not using a chroot, Solr throws an exception starting up telling you to use a chroot if you want to use zk/solr security. I did this simply because I'm worried about how easy it is to not actually have security working correctly at the root level.

        Everything is working as expected AFAIK though.

        My only concern is reviewing any issues around loading custom imlpls - as we don't use the SolrResourceLoader, I just need to try it out manually and fully understand any implications of that. I understand SolrResourceLoader cannot easily be used because these custom impls are necessary for solrj clients and such, I just want to test and review for my own peace of mind.

        Show
        Mark Miller added a comment - If you try and use zk security without a chroot, it fails to start. I realize I may have been unclear here. It fails to start because I added code to make it so - if you attempt to configure zk/solr security and are not using a chroot, Solr throws an exception starting up telling you to use a chroot if you want to use zk/solr security. I did this simply because I'm worried about how easy it is to not actually have security working correctly at the root level. Everything is working as expected AFAIK though. My only concern is reviewing any issues around loading custom imlpls - as we don't use the SolrResourceLoader, I just need to try it out manually and fully understand any implications of that. I understand SolrResourceLoader cannot easily be used because these custom impls are necessary for solrj clients and such, I just want to test and review for my own peace of mind.
        Hide
        Mark Miller added a comment -

        It fails to start because I added code to make it so

        I pulled that out. The complication is not worth it. We should just doc the issue.

        I've changed it so that Solr itself uses the CoreContainer resource loader to load the impls using standard CoreContainer config. You can still use the sys props for things like CloudSolrServer and ZkCLI.

        Show
        Mark Miller added a comment - It fails to start because I added code to make it so I pulled that out. The complication is not worth it. We should just doc the issue. I've changed it so that Solr itself uses the CoreContainer resource loader to load the impls using standard CoreContainer config. You can still use the sys props for things like CloudSolrServer and ZkCLI.
        Hide
        ASF subversion and git services added a comment -

        Commit 1621294 from Mark Miller in branch 'dev/trunk'
        [ https://svn.apache.org/r1621294 ]

        SOLR-4580: Support for protecting content in ZooKeeper.

        Show
        ASF subversion and git services added a comment - Commit 1621294 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1621294 ] SOLR-4580 : Support for protecting content in ZooKeeper.
        Hide
        ASF subversion and git services added a comment -

        Commit 1621295 from Mark Miller in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1621295 ]

        SOLR-4580: Support for protecting content in ZooKeeper.

        Show
        ASF subversion and git services added a comment - Commit 1621295 from Mark Miller in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1621295 ] SOLR-4580 : Support for protecting content in ZooKeeper.
        Hide
        Mark Miller added a comment -

        Thanks Per!

        Show
        Mark Miller added a comment - Thanks Per!
        Hide
        Shalin Shekhar Mangar added a comment -

        This has broken the start scripts as well as the cloud-dev scripts. On starting solr using ./bin/solr -c, I see the following in the logs:

        2204 [main] ERROR org.apache.solr.servlet.SolrDispatchFilter – Could not start Solr. Check solr/home property and the logs
        2224 [main] ERROR org.apache.solr.core.SolrCore – null:java.lang.StringIndexOutOfBoundsException: String index out of range: -1
        at java.lang.String.substring(String.java:1954)
        at org.apache.solr.core.ZkContainer.stripChroot(ZkContainer.java:199)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:104)
        at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:64)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:222)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:190)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:137)
        at org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:119)

        Show
        Shalin Shekhar Mangar added a comment - This has broken the start scripts as well as the cloud-dev scripts. On starting solr using ./bin/solr -c, I see the following in the logs: 2204 [main] ERROR org.apache.solr.servlet.SolrDispatchFilter – Could not start Solr. Check solr/home property and the logs 2224 [main] ERROR org.apache.solr.core.SolrCore – null:java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1954) at org.apache.solr.core.ZkContainer.stripChroot(ZkContainer.java:199) at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:104) at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:64) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:222) at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:190) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:137) at org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:119)
        Hide
        ASF subversion and git services added a comment -

        Commit 1622319 from Mark Miller in branch 'dev/trunk'
        [ https://svn.apache.org/r1622319 ]

        SOLR-4580: fix strip chroot call.

        Show
        ASF subversion and git services added a comment - Commit 1622319 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1622319 ] SOLR-4580 : fix strip chroot call.
        Hide
        ASF subversion and git services added a comment -

        Commit 1622325 from Mark Miller in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1622325 ]

        SOLR-4580: fix strip chroot call.

        Show
        ASF subversion and git services added a comment - Commit 1622325 from Mark Miller in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1622325 ] SOLR-4580 : fix strip chroot call.
        Hide
        Mark Miller added a comment -

        Erick just mentioned he noticed we still have to deal with the case there is no trailing slash.

        Show
        Mark Miller added a comment - Erick just mentioned he noticed we still have to deal with the case there is no trailing slash.
        Hide
        Erick Erickson added a comment -

        There's another problem here with the strip chroot call, I'm testing a fix now, will commit shortly if it works.

        Show
        Erick Erickson added a comment - There's another problem here with the strip chroot call, I'm testing a fix now, will commit shortly if it works.
        Hide
        ASF subversion and git services added a comment -

        Commit 1622339 from Erick Erickson in branch 'dev/trunk'
        [ https://svn.apache.org/r1622339 ]

        SOLR-4580: fix a second problem with stripChroot call

        Show
        ASF subversion and git services added a comment - Commit 1622339 from Erick Erickson in branch 'dev/trunk' [ https://svn.apache.org/r1622339 ] SOLR-4580 : fix a second problem with stripChroot call
        Hide
        ASF subversion and git services added a comment -

        Commit 1622340 from Erick Erickson in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1622340 ]

        SOLR-4580: fix a second problem with stripChroot call

        Show
        ASF subversion and git services added a comment - Commit 1622340 from Erick Erickson in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1622340 ] SOLR-4580 : fix a second problem with stripChroot call
        Hide
        Erick Erickson added a comment -

        Is this right yet? If we have a root like a/b/c/ we'd get a return of a/b/c. OK, that's good
        but
        a/b/c would return a/b

        Or is multi-level root not possible in this case?

        Show
        Erick Erickson added a comment - Is this right yet? If we have a root like a/b/c/ we'd get a return of a/b/c. OK, that's good but a/b/c would return a/b Or is multi-level root not possible in this case?
        Hide
        Mark Miller added a comment -

        Whoops, firefox playing tricks on me.

        Show
        Mark Miller added a comment - Whoops, firefox playing tricks on me.
        Hide
        Anshum Gupta added a comment -

        Bulk close after 5.0 release.

        Show
        Anshum Gupta added a comment - Bulk close after 5.0 release.

          People

          • Assignee:
            Mark Miller
            Reporter:
            Per Steffensen
          • Votes:
            3 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development