Details

      Description

      The documentation presented here: https://cwiki.apache.org/confluence/display/solr/ZooKeeper+Access+Control
      details the process of securing Solr content in ZooKeeper using ACLs. In the example usages, it is mentioned that access to zkcli can be restricted by adding credentials to the zkcli.sh script in addition to adding the appropriate classnames to solr.xml. With the scripts in zkcli.sh, another machine should not be able to read or write from the host ZK without the necessary credentials. At this time, machines are able to read/write from the host ZK with or without these credentials.

      1. SOLR-8792.patch
        12 kB
        Steve Rowe
      2. SOLR-8792.patch
        12 kB
        Steve Rowe
      3. SOLR-8792.patch
        11 kB
        Steve Rowe
      4. SOLR-8792.patch
        1 kB
        Ishan Chattopadhyaya

        Activity

        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        It seems that during the Solr startup, the ACL provider is already chosen (as the default) even before solr.xml is loaded. Hence, specifying a different ACL provider is not working.

        INFO  - 2016-04-14 08:15:59.039; [   ] org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsProvider
        INFO  - 2016-04-14 08:15:59.059; [   ] org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
        INFO  - 2016-04-14 08:15:59.145; [   ] org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@30b99be5 name:ZooKeeperConnection Watcher:zk1:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
        INFO  - 2016-04-14 08:15:59.145; [   ] org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
        INFO  - 2016-04-14 08:15:59.145; [   ] org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider
        INFO  - 2016-04-14 08:15:59.160; [   ] org.apache.solr.servlet.SolrDispatchFilter; Loading solr.xml from SolrHome (not found in ZooKeeper)
        
        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - It seems that during the Solr startup, the ACL provider is already chosen (as the default) even before solr.xml is loaded. Hence, specifying a different ACL provider is not working. INFO - 2016-04-14 08:15:59.039; [ ] org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsProvider INFO - 2016-04-14 08:15:59.059; [ ] org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper INFO - 2016-04-14 08:15:59.145; [ ] org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@30b99be5 name:ZooKeeperConnection Watcher:zk1:2181 got event WatchedEvent state:SyncConnected type:None path: null path: null type:None INFO - 2016-04-14 08:15:59.145; [ ] org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper INFO - 2016-04-14 08:15:59.145; [ ] org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider INFO - 2016-04-14 08:15:59.160; [ ] org.apache.solr.servlet.SolrDispatchFilter; Loading solr.xml from SolrHome (not found in ZooKeeper)
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Actually, that default ACL provider log message was misleading. It is not for the default ZK client, but only for a short lived client that is used to fetch the solr.xml from ZK (if present). I can see that the VM params based ACL provider is kicking in.

        However, in my efforts to have SolrCLI work, now I am dealing with another problem: SolrCLI doesn't get the ZK username/password passed in by the bin/solr script while creating collections, and hence collection creation is failing. Looking into that for now.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Actually, that default ACL provider log message was misleading. It is not for the default ZK client, but only for a short lived client that is used to fetch the solr.xml from ZK (if present). I can see that the VM params based ACL provider is kicking in. However, in my efforts to have SolrCLI work, now I am dealing with another problem: SolrCLI doesn't get the ZK username/password passed in by the bin/solr script while creating collections, and hence collection creation is failing. Looking into that for now.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment - - edited

        Finally managed to have it work for me. Attached the patch for a fix to bin/solr script.

        Here are the steps that need to be taken to enable ZK ACLs (assuming the attached patch is committed or applied).

        1. Start a fresh ZK instance (lets assume the hostname is zk1).
        2. Add the following to the ./server/solr/solr.xml file's SolrCloud section:
          <str name="zkCredentialsProvider">org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider</str>
          <str name="zkACLProvider">org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider</str>
          
        3. In the bin/solr.in.sh's section called Settings for ZK ACL, provide all the passwords etc. (one could uncomment and edit the existing lines there). Here's an example:
          # Settings for ZK ACL
          SOLR_ZK_CREDS_AND_ACLS="-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider"
          SOLR_ZK_CREDS_AND_ACLS="$SOLR_ZK_CREDS_AND_ACLS -DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider"
          SOLR_ZK_CREDS_AND_ACLS="$SOLR_ZK_CREDS_AND_ACLS -DzkDigestUsername=admin-user -DzkDigestPassword=admin-password"
          SOLR_ZK_CREDS_AND_ACLS="$SOLR_ZK_CREDS_AND_ACLS -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=readonly-password"
          SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"
          
        4. Start Solr. bin/solr start -e cloud -z zk1:2181 -noprompt
        5. To ensure that unauthorized access is restricted, try uploading a configset to ZK (without any changes to the zkcli.sh):
          server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -confname basic -z 
          zk1:2181 -confdir server/solr/configsets/basic_configs
          

          This should result in an error:

          Exception in thread "main" java.io.IOException: Error uploading file server/solr/configsets/basic_configs/conf/_rest_managed.json to zookeeper path /configs/basic/conf/_rest_managed.json
          	at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:78)
          	at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:65)
          	at java.nio.file.Files.walkFileTree(Files.java:2670)
          	at java.nio.file.Files.walkFileTree(Files.java:2742)
          	at org.apache.solr.common.cloud.ZkConfigManager.uploadToZK(ZkConfigManager.java:65)
          	at org.apache.solr.common.cloud.ZkConfigManager.uploadConfigDir(ZkConfigManager.java:142)
          	at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:227)
          Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /configs/basic
          	at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
          	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
          	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
          	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:503)
          	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
          	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:500)
          	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411)
          	at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:75)
          	... 6 more
          
          
        6. To have zkcli.sh work, add the credentials to zkcli.sh
          
          SOLR_ZK_CREDS_AND_ACLS="-DzkDigestUsername=admin-user -DzkDigestPassword=admin-password \
          -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=readonly-password"
           
          java ... $SOLR_ZK_CREDS_AND_ACLS ... org.apache.solr.cloud.ZkCLI -cmd ...
          
        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - - edited Finally managed to have it work for me. Attached the patch for a fix to bin/solr script. Here are the steps that need to be taken to enable ZK ACLs (assuming the attached patch is committed or applied). Start a fresh ZK instance (lets assume the hostname is zk1). Add the following to the ./server/solr/solr.xml file's SolrCloud section: <str name= "zkCredentialsProvider" >org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider</str> <str name= "zkACLProvider" >org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider</str> In the bin/solr.in.sh's section called Settings for ZK ACL , provide all the passwords etc. (one could uncomment and edit the existing lines there). Here's an example: # Settings for ZK ACL SOLR_ZK_CREDS_AND_ACLS= "-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider" SOLR_ZK_CREDS_AND_ACLS= "$SOLR_ZK_CREDS_AND_ACLS -DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider" SOLR_ZK_CREDS_AND_ACLS= "$SOLR_ZK_CREDS_AND_ACLS -DzkDigestUsername=admin-user -DzkDigestPassword=admin-password" SOLR_ZK_CREDS_AND_ACLS= "$SOLR_ZK_CREDS_AND_ACLS -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=readonly-password" SOLR_OPTS= "$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS" Start Solr. bin/solr start -e cloud -z zk1:2181 -noprompt To ensure that unauthorized access is restricted, try uploading a configset to ZK (without any changes to the zkcli.sh): server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -confname basic -z zk1:2181 -confdir server/solr/configsets/basic_configs This should result in an error: Exception in thread "main" java.io.IOException: Error uploading file server/solr/configsets/basic_configs/conf/_rest_managed.json to zookeeper path /configs/basic/conf/_rest_managed.json at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:78) at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:65) at java.nio.file.Files.walkFileTree(Files.java:2670) at java.nio.file.Files.walkFileTree(Files.java:2742) at org.apache.solr.common.cloud.ZkConfigManager.uploadToZK(ZkConfigManager.java:65) at org.apache.solr.common.cloud.ZkConfigManager.uploadConfigDir(ZkConfigManager.java:142) at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:227) Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /configs/basic at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:503) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:500) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411) at org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:75) ... 6 more To have zkcli.sh work, add the credentials to zkcli.sh SOLR_ZK_CREDS_AND_ACLS="-DzkDigestUsername=admin-user -DzkDigestPassword=admin-password \ -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=readonly-password" java ... $SOLR_ZK_CREDS_AND_ACLS ... org.apache.solr.cloud.ZkCLI -cmd ...
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Can someone please review and commit this? Without this fix, ACL support is broken from user perspective. I think this should also be backported to 5x, and possibly be included in 5.5.1.

        Here's a video demonstration of the fix and the steps above (which should be documented in the ref guide): https://www.youtube.com/watch?v=Sl9R_cVI27o

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Can someone please review and commit this? Without this fix, ACL support is broken from user perspective. I think this should also be backported to 5x, and possibly be included in 5.5.1. Here's a video demonstration of the fix and the steps above (which should be documented in the ref guide): https://www.youtube.com/watch?v=Sl9R_cVI27o
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Adding the fix version to 5.5.1. Please remove if someone thinks that is inappropriate.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Adding the fix version to 5.5.1. Please remove if someone thinks that is inappropriate.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Maybe the boat has already sailed for 5.5.1, but I would still like to bring this issue to your attention, Anshum Gupta.
        At this point, I think this issue to land in 5.5.1 looks difficult since this has not been reviewed yet, but I still believe this issue is critical enough to be fixed in some 5x version.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Maybe the boat has already sailed for 5.5.1, but I would still like to bring this issue to your attention, Anshum Gupta . At this point, I think this issue to land in 5.5.1 looks difficult since this has not been reviewed yet, but I still believe this issue is critical enough to be fixed in some 5x version.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Removing fix version 5.5.1.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Removing fix version 5.5.1.
        Hide
        steve_rowe Steve Rowe added a comment -

        Patch adding support for Windows and Solr's zkcli scripts.

        I'll do some manual testing before I commit.

        Show
        steve_rowe Steve Rowe added a comment - Patch adding support for Windows and Solr's zkcli scripts. I'll do some manual testing before I commit.
        Hide
        steve_rowe Steve Rowe added a comment -

        Couple changes from previous patch:

        1. server/solr/solr.xml now includes zkACLProvider and zkCredentialsProvider config in the <solrcloud> section that pull in the corresponding sysprops if defined, and defaults to the default implementations if not. This way the user doesn't have to modify solr.xml at all.
        2. The zkcli script additions are corrected to include zkACLProvider and zkCredentialsProvider sysprops (previously didn't include them).
        3. Passwords in commented out ZK ACL config are now CHANGEME-ADMIN-PASSWORD and CHANGEME-READONLY-PASSWORD (previously were admin-password and readonly-password).

        Manual testing on OS X looks good, will do some Windows testing before I commit.

        Show
        steve_rowe Steve Rowe added a comment - Couple changes from previous patch: server/solr/solr.xml now includes zkACLProvider and zkCredentialsProvider config in the <solrcloud> section that pull in the corresponding sysprops if defined, and defaults to the default implementations if not. This way the user doesn't have to modify solr.xml at all. The zkcli script additions are corrected to include zkACLProvider and zkCredentialsProvider sysprops (previously didn't include them). Passwords in commented out ZK ACL config are now CHANGEME-ADMIN-PASSWORD and CHANGEME-READONLY-PASSWORD (previously were admin-password and readonly-password ). Manual testing on OS X looks good, will do some Windows testing before I commit.
        Hide
        steve_rowe Steve Rowe added a comment -

        Final patch, removes extra quotes in Windows scripts and adds a CHANGES entry.

        With this patch, ACLs work properly on Windows 7, but I couldn't get the cloud example to run with bin\solr.cmd - it prints the command to start node1 but then can't see it after 30 seconds, and no logs are produced. I'll investigate that separately. Manually starting the individual nodes and creating a collection worked fine with bin\solr.cmd. Annoyingly, the environment variable set in zkcli.bat leaks into the dos box environment, so in order to test that it couldn't access ZK after Solr had set restricted ACLs on content, from the cmdline I had to first unset the environment variable used to specify ZK ACL and credentials provider classes, user names and passwords.

        Committing shortly.

        Show
        steve_rowe Steve Rowe added a comment - Final patch, removes extra quotes in Windows scripts and adds a CHANGES entry. With this patch, ACLs work properly on Windows 7, but I couldn't get the cloud example to run with bin\solr.cmd - it prints the command to start node1 but then can't see it after 30 seconds, and no logs are produced. I'll investigate that separately. Manually starting the individual nodes and creating a collection worked fine with bin\solr.cmd . Annoyingly, the environment variable set in zkcli.bat leaks into the dos box environment, so in order to test that it couldn't access ZK after Solr had set restricted ACLs on content, from the cmdline I had to first unset the environment variable used to specify ZK ACL and credentials provider classes, user names and passwords. Committing shortly.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 5d4cd44b6dc4d23ff90c0536371c9db0701a304a in lucene-solr's branch refs/heads/master from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5d4cd44 ]

        SOLR-8792: ZooKeeper ACL support fixed

        Show
        jira-bot ASF subversion and git services added a comment - Commit 5d4cd44b6dc4d23ff90c0536371c9db0701a304a in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5d4cd44 ] SOLR-8792 : ZooKeeper ACL support fixed
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a5e2be0444801f2e94683007d1fa5c3c893765fd in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a5e2be0 ]

        SOLR-8792: ZooKeeper ACL support fixed

        Show
        jira-bot ASF subversion and git services added a comment - Commit a5e2be0444801f2e94683007d1fa5c3c893765fd in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a5e2be0 ] SOLR-8792 : ZooKeeper ACL support fixed
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        We should really look at how to start testing this type of thing better longer term. It would be great if things like bat scripts were tested when run on the Policeman Windows box for example, and sh scripts on the LInux runs. Even if it was just some simple unit tests that shell out.

        Show
        markrmiller@gmail.com Mark Miller added a comment - We should really look at how to start testing this type of thing better longer term. It would be great if things like bat scripts were tested when run on the Policeman Windows box for example, and sh scripts on the LInux runs. Even if it was just some simple unit tests that shell out.
        Hide
        steve_rowe Steve Rowe added a comment -

        Reopening to backport to 6.0.1.

        Show
        steve_rowe Steve Rowe added a comment - Reopening to backport to 6.0.1.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1af2907eee045c487b6c8a5c7676eed6b56234dd in lucene-solr's branch refs/heads/branch_6_0 from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1af2907 ]

        SOLR-8792: ZooKeeper ACL support fixed

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1af2907eee045c487b6c8a5c7676eed6b56234dd in lucene-solr's branch refs/heads/branch_6_0 from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1af2907 ] SOLR-8792 : ZooKeeper ACL support fixed
        Hide
        steve_rowe Steve Rowe added a comment -

        Bulk close issues included in the 6.0.1 release.

        Show
        steve_rowe Steve Rowe added a comment - Bulk close issues included in the 6.0.1 release.
        Hide
        steve_rowe Steve Rowe added a comment -

        Reopening to backport to 5.6 and 5.5.2.

        Show
        steve_rowe Steve Rowe added a comment - Reopening to backport to 5.6 and 5.5.2.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 41e1a9061fcc2e7a3a93724845e313e880726ceb in lucene-solr's branch refs/heads/branch_5_5 from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=41e1a90 ]

        SOLR-8792: ZooKeeper ACL support fixed

        Show
        jira-bot ASF subversion and git services added a comment - Commit 41e1a9061fcc2e7a3a93724845e313e880726ceb in lucene-solr's branch refs/heads/branch_5_5 from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=41e1a90 ] SOLR-8792 : ZooKeeper ACL support fixed
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 9a30f091b657c9775738f3df29a5f6cc37495bc2 in lucene-solr's branch refs/heads/branch_5x from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9a30f09 ]

        SOLR-8792: ZooKeeper ACL support fixed

        Show
        jira-bot ASF subversion and git services added a comment - Commit 9a30f091b657c9775738f3df29a5f6cc37495bc2 in lucene-solr's branch refs/heads/branch_5x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9a30f09 ] SOLR-8792 : ZooKeeper ACL support fixed
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 7f721752eef8560c9c3e03b7680599d7cbb2c7fa in lucene-solr's branch refs/heads/branch_5x from Steve Rowe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7f72175 ]

        SOLR-8792: Remove misplaced CHANGES entry

        Show
        jira-bot ASF subversion and git services added a comment - Commit 7f721752eef8560c9c3e03b7680599d7cbb2c7fa in lucene-solr's branch refs/heads/branch_5x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7f72175 ] SOLR-8792 : Remove misplaced CHANGES entry
        Hide
        steve_rowe Steve Rowe added a comment -

        Bulk close issues released with 5.5.2.

        Show
        steve_rowe Steve Rowe added a comment - Bulk close issues released with 5.5.2.

          People

          • Assignee:
            steve_rowe Steve Rowe
            Reporter:
            esther.quansah Esther Quansah
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development