Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9516

New UI doesn't work when Kerberos is enabled

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, 7.0
    • Component/s: Admin UI
    • Security Level: Public (Default Security Level. Issues are Public)

      Description

      It seems resources like http://solr1:8983/solr/libs/chosen.jquery.js encounter 403 error:

      2016-09-15 02:01:45.272 WARN  (qtp611437735-18) [   ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))
      

      The old UI is fine.

      1. QQ20161012-0.png
        1.03 MB
        loushang
      2. Screenshot from 2016-09-15 07-36-29.png
        70 kB
        Ishan Chattopadhyaya
      3. SOLR-9516.patch
        0.4 kB
        Amrit Sarkar

        Activity

        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Here's a screenshot from Firefox's Web Console (Network tab) indicating the some of the resources that exhibited this problem.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Here's a screenshot from Firefox's Web Console (Network tab) indicating the some of the resources that exhibited this problem.
        Hide
        arafalov Alexandre Rafalovitch added a comment -

        Which version of Solr is this reported against?

        Show
        arafalov Alexandre Rafalovitch added a comment - Which version of Solr is this reported against?
        Hide
        loushang loushang added a comment - - edited

        i get the same problem now. the solr version is 5.5.2

        see the QQ20161012-0.png in the attachment

        Show
        loushang loushang added a comment - - edited i get the same problem now. the solr version is 5.5.2 see the QQ20161012-0.png in the attachment
        Hide
        arafalov Alexandre Rafalovitch added a comment -

        Could you try that against Solr 6.2? Because there had been a large number of issues fixed both for UI and for various security components.

        Show
        arafalov Alexandre Rafalovitch added a comment - Could you try that against Solr 6.2? Because there had been a large number of issues fixed both for UI and for various security components.
        Hide
        ctargett Cassandra Targett added a comment -

        I believe Ishan Chattopadhyaya was using 6.2 when he found the problem.

        Show
        ctargett Cassandra Targett added a comment - I believe Ishan Chattopadhyaya was using 6.2 when he found the problem.
        Hide
        loushang loushang added a comment -

        so you confirmed that this bug was not fixed in 6.2?

        Show
        loushang loushang added a comment - so you confirmed that this bug was not fixed in 6.2?
        Hide
        ctargett Cassandra Targett added a comment -

        If the plan is to remove the old Admin UI for Solr 7, then this needs to be considered a blocker for that.

        Show
        ctargett Cassandra Targett added a comment - If the plan is to remove the old Admin UI for Solr 7, then this needs to be considered a blocker for that.
        Hide
        arafalov Alexandre Rafalovitch added a comment -

        It would be a blocker if it worked in old UI and we had it replicated on the latest Solr. At the moment, we don't have either fact confirmed, nor do we have the replication instructions.

        I am not familiar with Kerberos setup, but if somebody provides replication instructions, I'll be happy to dig into it.

        Show
        arafalov Alexandre Rafalovitch added a comment - It would be a blocker if it worked in old UI and we had it replicated on the latest Solr. At the moment, we don't have either fact confirmed, nor do we have the replication instructions. I am not familiar with Kerberos setup, but if somebody provides replication instructions, I'll be happy to dig into it.
        Hide
        ctargett Cassandra Targett added a comment -

        Replication is pretty straightforward if you have Kerberos already in your env:

        • Setup Solr to use Kerberos authentication.
        • Try to use the admin UI. You can't, with the 403 errors as seen in previously attached screenshots.

        The old Admin UI works fine.

        The rub here is that this requires setting up a Kerberos ticket server, etc., which really isn't the easiest thing in the world (I don't know how to do it, for example).

        Lucidworks has customers who have seen this, most recently using 6.4.1, so I can confirm it still exists in the latest Solr release. Ishan's initial report was against 6.2, so it's not just the latest; IMO we can assume it's all 6.x releases, and maybe even 5.5.2 per another reporter.

        There's a workaround for now - use the old UI - but if the old UI is removed, users with Kerberos auth will be unable to use the Admin UI.

        Show
        ctargett Cassandra Targett added a comment - Replication is pretty straightforward if you have Kerberos already in your env: Setup Solr to use Kerberos authentication. Try to use the admin UI. You can't, with the 403 errors as seen in previously attached screenshots. The old Admin UI works fine. The rub here is that this requires setting up a Kerberos ticket server, etc., which really isn't the easiest thing in the world (I don't know how to do it, for example). Lucidworks has customers who have seen this, most recently using 6.4.1, so I can confirm it still exists in the latest Solr release. Ishan's initial report was against 6.2, so it's not just the latest; IMO we can assume it's all 6.x releases, and maybe even 5.5.2 per another reporter. There's a workaround for now - use the old UI - but if the old UI is removed, users with Kerberos auth will be unable to use the Admin UI.
        Hide
        arafalov Alexandre Rafalovitch added a comment -

        I am looking at: https://cwiki.apache.org/confluence/display/solr/Kerberos+Authentication+Plugin
        It says:

        In order for your browser to access the Solr Admin UI after enabling Kerberos authentication, it must be able to negotiate with the Kerberos authenticator service to allow you access. Each browser supports this differently, and some (like Chrome) do not support it at all. If you see 401 errors when trying to access the Solr Admin UI after enabling Kerberos authentication, it's likely your browser has not been configured properly to know how or where to negotiate the authentication request.

        Detailed information on how to set up your browser is beyond the scope of this documentation; please see your system administrators for Kerberos for details on how to configure your browser.

        Are we - absolutely - sure that the exact same setup works with the old UI? Could we get full browser/network traces for a request made from old UI and from New UI? Preferably while the backend is actually running with full TRACE log.

        Show
        arafalov Alexandre Rafalovitch added a comment - I am looking at: https://cwiki.apache.org/confluence/display/solr/Kerberos+Authentication+Plugin It says: In order for your browser to access the Solr Admin UI after enabling Kerberos authentication, it must be able to negotiate with the Kerberos authenticator service to allow you access. Each browser supports this differently, and some (like Chrome) do not support it at all. If you see 401 errors when trying to access the Solr Admin UI after enabling Kerberos authentication, it's likely your browser has not been configured properly to know how or where to negotiate the authentication request. Detailed information on how to set up your browser is beyond the scope of this documentation; please see your system administrators for Kerberos for details on how to configure your browser. Are we - absolutely - sure that the exact same setup works with the old UI? Could we get full browser/network traces for a request made from old UI and from New UI? Preferably while the backend is actually running with full TRACE log.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        I can confirm that the exact setup works fine with old UI, but doesn't work with new UI. I'll reproduce and try to post logs. (When I saw this last time, I had no idea how to even copy logs)

        Amrit Sarkar, would you have a chance to have a look at this issue, please? Given that you're actively working on the UI these days, and given my limited UI knowledge, I might need your help here.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - I can confirm that the exact setup works fine with old UI, but doesn't work with new UI. I'll reproduce and try to post logs. (When I saw this last time, I had no idea how to even copy logs) Amrit Sarkar , would you have a chance to have a look at this issue, please? Given that you're actively working on the UI these days, and given my limited UI knowledge, I might need your help here.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        I am not familiar with Kerberos setup, but if somebody provides replication instructions, I'll be happy to dig into it.

        https://github.com/chatman/solr-kerberos-docker

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - I am not familiar with Kerberos setup, but if somebody provides replication instructions, I'll be happy to dig into it. https://github.com/chatman/solr-kerberos-docker
        Hide
        sarkaramrit2@gmail.com Amrit Sarkar added a comment - - edited

        Ishan, sorry I didn't respond earlier, didn't notice the mention.

        http://host:port/solr/libs was inaccessible as it was not listed in exclusion pattern for SolrDispatchFilter, hence it required authentication and UI failed to fetch the content from that path from webapp folder. Thank you Cassandra Targett for pin-pointing the above and suggesting the changes.

        We faced similar Kerberos 34 Request is a Replay error for MBeans Request Handler:

        http://localhost:8983/solr/[collection_name]/admin/mbeans?cat=CACHE

        and the changes listed below rectified that, not sure why it was broken and thus how it got fixed.

        SOLR-9516.patch uploaded with one line change in web.xml in webapp.

        Show
        sarkaramrit2@gmail.com Amrit Sarkar added a comment - - edited Ishan, sorry I didn't respond earlier, didn't notice the mention. http://host:port/solr/libs was inaccessible as it was not listed in exclusion pattern for SolrDispatchFilter, hence it required authentication and UI failed to fetch the content from that path from webapp folder. Thank you Cassandra Targett for pin-pointing the above and suggesting the changes. We faced similar Kerberos 34 Request is a Replay error for MBeans Request Handler: http: //localhost:8983/solr/[collection_name]/admin/mbeans?cat=CACHE and the changes listed below rectified that, not sure why it was broken and thus how it got fixed. SOLR-9516 .patch uploaded with one line change in web.xml in webapp.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Amrit, I think it is fine to exclude the /solr/libs from being served through SDF (and hence leaving them unauthenticated), but that still doesn't explain why the authentication was failing for them. Excluding them from authentication makes sense due to performance reasons, but this is just a workaround to some other potential problem with SDF/Kerberos.

        If you're sure that other parts of UI also work fine (collections/core admin command buttons, zk/file tree etc.), then lets just commit the patch now and worry later as to why authentication wasn't working for /solr/lib endpoints (there could be some other problem with the way SDF works in certain cases).

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Amrit, I think it is fine to exclude the /solr/libs from being served through SDF (and hence leaving them unauthenticated), but that still doesn't explain why the authentication was failing for them. Excluding them from authentication makes sense due to performance reasons, but this is just a workaround to some other potential problem with SDF/Kerberos. If you're sure that other parts of UI also work fine (collections/core admin command buttons, zk/file tree etc.), then lets just commit the patch now and worry later as to why authentication wasn't working for /solr/lib endpoints (there could be some other problem with the way SDF works in certain cases).
        Hide
        sarkaramrit2@gmail.com Amrit Sarkar added a comment -

        Ishan, all the buttons, commands, stats, tree, cloud, thread info, dashboard are working as expected positively.

        Show
        sarkaramrit2@gmail.com Amrit Sarkar added a comment - Ishan, all the buttons, commands, stats, tree, cloud, thread info, dashboard are working as expected positively.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 65c695b025ad0efb952494f767c1ec9fa44a4924 in lucene-solr's branch refs/heads/master from Ishan Chattopadhyaya
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=65c695b ]

        SOLR-9516: Fix: Admin UI (angular) didn't work with Kerberos

        Show
        jira-bot ASF subversion and git services added a comment - Commit 65c695b025ad0efb952494f767c1ec9fa44a4924 in lucene-solr's branch refs/heads/master from Ishan Chattopadhyaya [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=65c695b ] SOLR-9516 : Fix: Admin UI (angular) didn't work with Kerberos
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 46de138214169e13162a74be46d8fedfd508d98a in lucene-solr's branch refs/heads/branch_6x from Ishan Chattopadhyaya
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=46de138 ]

        SOLR-9516: Fix: Admin UI (angular) didn't work with Kerberos

        Show
        jira-bot ASF subversion and git services added a comment - Commit 46de138214169e13162a74be46d8fedfd508d98a in lucene-solr's branch refs/heads/branch_6x from Ishan Chattopadhyaya [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=46de138 ] SOLR-9516 : Fix: Admin UI (angular) didn't work with Kerberos
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -
        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Thanks Cassandra Targett , Amrit Sarkar !
        Hide
        varunthacker Varun Thacker added a comment -

        Hi Ishan,

        Small nit:

        Can we change the CHANGES entry from

        SOLR-9516: Admin UI (angular) didn't work with Kerberos (Cassandra Targett, Amrit Sarkar via Ishan Chattopadhyaya)
        

        to something like this? Just a suggestion based on what I learnt from https://www.mail-archive.com/dev@lucene.apache.org/msg144137.html

        SOLR-9516: Admin UI now works with Kerberos (Cassandra Targett, Amrit Sarkar via Ishan Chattopadhyaya)
        
        Show
        varunthacker Varun Thacker added a comment - Hi Ishan, Small nit: Can we change the CHANGES entry from SOLR-9516: Admin UI (angular) didn't work with Kerberos (Cassandra Targett, Amrit Sarkar via Ishan Chattopadhyaya) to something like this? Just a suggestion based on what I learnt from https://www.mail-archive.com/dev@lucene.apache.org/msg144137.html SOLR-9516: Admin UI now works with Kerberos (Cassandra Targett, Amrit Sarkar via Ishan Chattopadhyaya)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 2bce98b0c162c5d8a815bc3e2ec32ba6d08c62fa in lucene-solr's branch refs/heads/master from Ishan Chattopadhyaya
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2bce98b ]

        SOLR-9516: Updating CHANGES.txt entry

        Show
        jira-bot ASF subversion and git services added a comment - Commit 2bce98b0c162c5d8a815bc3e2ec32ba6d08c62fa in lucene-solr's branch refs/heads/master from Ishan Chattopadhyaya [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2bce98b ] SOLR-9516 : Updating CHANGES.txt entry
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Thanks Varun. Pushed to branch_6x as per commit c6a1aa20dc9f2e0bffcdb42cf01efff6466b1128.

        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c6a1aa20dc9f2e0bffcdb42cf01efff6466b1128 ]

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Thanks Varun. Pushed to branch_6x as per commit c6a1aa20dc9f2e0bffcdb42cf01efff6466b1128. [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c6a1aa20dc9f2e0bffcdb42cf01efff6466b1128 ]

          People

          • Assignee:
            ichattopadhyaya Ishan Chattopadhyaya
            Reporter:
            ichattopadhyaya Ishan Chattopadhyaya
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development