Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      Add Knox support for routing and securing Apache SOLR's REST APIs

      See: https://wiki.apache.org/solr/Solrj

      1. KNOX-528_basicAuth_urlParamsNoOrder_squashed.patch
        9 kB
        John McParland
      2. KNOX-528_basicauth_squashed.patch
        7 kB
        John McParland
      3. KNOX-528_squashed.patch
        7 kB
        John McParland
      4. KNOX-528.patch
        49 kB
        John McParland
      5. knoxSolrTestEvidence.txt
        6 kB
        John McParland
      6. solrTestEvidence.txt
        6 kB
        John McParland

        Issue Links

          Activity

          Hide
          risdenk Kevin Risden added a comment -

          Just want to confirm that these rewrite rules only work for the REST API and don't have rules in there for the UI. It would be awesome to support the Solr UI as well. That could be another JIRA.

          Show
          risdenk Kevin Risden added a comment - Just want to confirm that these rewrite rules only work for the REST API and don't have rules in there for the UI. It would be awesome to support the Solr UI as well. That could be another JIRA.
          Hide
          mcparlandjcgi John McParland added a comment -
          Show
          mcparlandjcgi John McParland added a comment - Thanks Larry McCay , Kevin Risden and Sandeep More !
          Hide
          lmccay Larry McCay added a comment -

          Kevin Risden - I failed to thank you as well! Will make a note to make sure you get credit in the CHANGES file for the release!

          Show
          lmccay Larry McCay added a comment - Kevin Risden - I failed to thank you as well! Will make a note to make sure you get credit in the CHANGES file for the release!
          Hide
          risdenk Kevin Risden added a comment -

          Awesome! Glad to see this get in.

          Show
          risdenk Kevin Risden added a comment - Awesome! Glad to see this get in.
          Hide
          lmccay Larry McCay added a comment -

          John McParland and Sandeep More - I've committed this to master and it will be in 0.11.0 release.
          Thank you for your contributions to Knox they are much appreciated!

          Show
          lmccay Larry McCay added a comment - John McParland and Sandeep More - I've committed this to master and it will be in 0.11.0 release. Thank you for your contributions to Knox they are much appreciated!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d4ae9ae5220f1e2dc2a874592276856f35e54b3a in knox's branch refs/heads/master from Larry McCay
          [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=d4ae9ae ]

          KNOX-528 - Support for Apache SOLR REST APIs (John McParland via lmccay)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d4ae9ae5220f1e2dc2a874592276856f35e54b3a in knox's branch refs/heads/master from Larry McCay [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=d4ae9ae ] KNOX-528 - Support for Apache SOLR REST APIs (John McParland via lmccay)
          Hide
          smore Sandeep More added a comment -

          Thanks John McParland, the patch looks OK to me and the UnitTest runs as well !
          +1

          Show
          smore Sandeep More added a comment - Thanks John McParland , the patch looks OK to me and the UnitTest runs as well ! +1
          Hide
          mcparlandjcgi John McParland added a comment -

          Hi Larry McCay I've attached a new patch. I'd love to see this in 0.11.0 too.

          Show
          mcparlandjcgi John McParland added a comment - Hi Larry McCay I've attached a new patch. I'd love to see this in 0.11.0 too.
          Hide
          mcparlandjcgi John McParland added a comment -

          Updated (and hopefully final) patch

          • Basic authentication used
          • Unit test for UrlRewriteProcessor updated to ensure args are tested independent of order
          • All commits squashed
          Show
          mcparlandjcgi John McParland added a comment - Updated (and hopefully final) patch Basic authentication used Unit test for UrlRewriteProcessor updated to ensure args are tested independent of order All commits squashed
          Hide
          lmccay Larry McCay added a comment -

          Hi John McParland - thanks for the explanation. I did see that the unit test is using the correct URL.
          Are you planning to provide a new revision of the patch that fixes the dependency on the ordering of the arguments in the test?

          I'd like to get this in for 0.11.0.

          Show
          lmccay Larry McCay added a comment - Hi John McParland - thanks for the explanation. I did see that the unit test is using the correct URL. Are you planning to provide a new revision of the patch that fixes the dependency on the ordering of the arguments in the test? I'd like to get this in for 0.11.0.
          Hide
          mcparlandjcgi John McParland added a comment -

          Hi Larry McCay
          The URL I mentioned in my last comment was to demonstrate a Solr only query - i.e. what I expect Knox to re-write the URL to, after the Rewrite Processor has run.

          Indeed, query going to Knox looks like this

          https://hdp24sandbox:8443/gateway/sandbox/solr/ExampleCollection/select?q=*.*&wt=json&indent=true

          And I'd expect it to be re-written as (assuming solr is on the same machine)

          http://hdp24sandbox/solr/ExampleCollection/select?q=*.*&wt=json&indent=true

          The test which Sandeep More has found is failing is one I wrote to test that the re-write rules I created do re-write this way. So it's failing when asserting the re-written URL, rather than the one which is sent into Knox.

          Show
          mcparlandjcgi John McParland added a comment - Hi Larry McCay The URL I mentioned in my last comment was to demonstrate a Solr only query - i.e. what I expect Knox to re-write the URL to, after the Rewrite Processor has run. Indeed, query going to Knox looks like this https://hdp24sandbox:8443/gateway/sandbox/solr/ExampleCollection/select?q=*.*&wt=json&indent=true And I'd expect it to be re-written as (assuming solr is on the same machine) http://hdp24sandbox/solr/ExampleCollection/select?q=*.*&wt=json&indent=true The test which Sandeep More has found is failing is one I wrote to test that the re-write rules I created do re-write this way. So it's failing when asserting the re-written URL, rather than the one which is sent into Knox.
          Hide
          lmccay Larry McCay added a comment -

          John McParland - I must be being really thick headed on this...

          I'm trying to see how these calls are actually going through Knox. They seem to be going directly to the solr application. In my mind, a Knox URL would look something like:

          https://sandbox.hortonworks.com:8443/gateway/sandbox/solr/KnoxSolrIntegration/select?q=*.*&indent=true&wt=json

          What we have above is missing the "gateway/sandbox", isn't using https and using the default http port of 80. It seems to me these calls are going directly to solr - around the knox gateway.

          Show
          lmccay Larry McCay added a comment - John McParland - I must be being really thick headed on this... I'm trying to see how these calls are actually going through Knox. They seem to be going directly to the solr application. In my mind, a Knox URL would look something like: https://sandbox.hortonworks.com:8443/gateway/sandbox/solr/KnoxSolrIntegration/select?q=*.*&indent=true&wt=json What we have above is missing the "gateway/sandbox", isn't using https and using the default http port of 80. It seems to me these calls are going directly to solr - around the knox gateway.
          Hide
          mcparlandjcgi John McParland added a comment - - edited

          Thanks Sandeep More and Larry McCay

          I'll update the article - the use of "Knox" throughout is mostly my choice of name for a Solr collection. It is confusing though.

          In terms of the URL, it breaks down as follows

          http://sandbox.hortonworks.com/solr/KnoxSolrIntegration/select?q=*.*&indent=true&wt=json
          
          Portion Meaning
          http://sandbox.hortonworks.com/ Host URL
          solr/ Solr application web context
          KnoxSolrIntegration/ The Solr collection to be queried (variable - I expect a user to specify it)
          select The type of Solr query to execute
          ? Arguments to query follow this
          q=*.* The argument to the query, in this case it means select everything
          indent=true Indent the output
          wt=json Return json formatted string

          Ordering of everything after the

          ?

          is not significant, so I can modify the test to take account of that.

          The reason for chosing Solr 5.5.0 is that's what comes when one installs HDP Search on an HDP 2.4 sandbox. I've generally based what I do off of the 2.4 sandbox defaults, for ease of configuration management and testing. I do not believe there's an API change in later versions.

          Show
          mcparlandjcgi John McParland added a comment - - edited Thanks Sandeep More and Larry McCay I'll update the article - the use of "Knox" throughout is mostly my choice of name for a Solr collection. It is confusing though. In terms of the URL, it breaks down as follows http://sandbox.hortonworks.com/solr/KnoxSolrIntegration/select?q=*.*&indent=true&wt=json Portion Meaning http://sandbox.hortonworks.com/ Host URL solr/ Solr application web context KnoxSolrIntegration/ The Solr collection to be queried (variable - I expect a user to specify it) select The type of Solr query to execute ? Arguments to query follow this q=*.* The argument to the query, in this case it means select everything indent=true Indent the output wt=json Return json formatted string Ordering of everything after the ? is not significant, so I can modify the test to take account of that. The reason for chosing Solr 5.5.0 is that's what comes when one installs HDP Search on an HDP 2.4 sandbox. I've generally based what I do off of the 2.4 sandbox defaults, for ease of configuration management and testing. I do not believe there's an API change in later versions.
          Hide
          lmccay Larry McCay added a comment -

          Kevin Risden - that would be cool. Let me try and figure out how and where in the codebase we would want to do that.

          Show
          lmccay Larry McCay added a comment - Kevin Risden - that would be cool. Let me try and figure out how and where in the codebase we would want to do that.
          Hide
          lmccay Larry McCay added a comment -

          John McParland and Sandeep More - Can one of you explain the URLs that are being used in the unit test assertionError above?
          I don't see the gateway/topology-name pattern that I would expect.

          Same for the article for setting up solr - which may be okay if it is only for setting up solr but it does mention Knox a good bit in it.
          Still it doesn't seem to be making any calls to Knox.

          I am probably missing something...

          Show
          lmccay Larry McCay added a comment - John McParland and Sandeep More - Can one of you explain the URLs that are being used in the unit test assertionError above? I don't see the gateway/topology-name pattern that I would expect. Same for the article for setting up solr - which may be okay if it is only for setting up solr but it does mention Knox a good bit in it. Still it doesn't seem to be making any calls to Knox. I am probably missing something...
          Hide
          smore Sandeep More added a comment -

          I ran a quick test on the patch and was able to get it work with just service definitions in sandbox.xml, I also ran it with HA configuration and works great, thanks John McParland good stuff !

          Some comments on the patch:
          1. The Unit Test failed for me, this was just because Knox shuffled the query arguments around

          java.lang.AssertionError: 
           Expected: is "http://sandbox.hortonworks.com/solr/KnoxSolrIntegration/select?q=*.*&indent=true&wt=json"
               but: was "http://sandbox.hortonworks.com/solr/KnoxSolrIntegration/select?indent=true&q=*.*&wt=json"
          

          Maybe break down the 'outputUri' into host/path/query and check - there might be other better ways to do this.

          2. Parser.parse( ) is deprecated so might be good to use Parser.parseTemplate() instead so that the test is not broken when the Parser.parse( ) api goes away.

          3. Solr latest release is 6.3.0 any reason why the version is 5.5.0. I know it does not matter as long as the api remains same, but it might give an impressions that the api is not tested (which, I did and works great !) and updated for 6.3.0.

          Again, thanks !

          Show
          smore Sandeep More added a comment - I ran a quick test on the patch and was able to get it work with just service definitions in sandbox.xml, I also ran it with HA configuration and works great, thanks John McParland good stuff ! Some comments on the patch: 1. The Unit Test failed for me, this was just because Knox shuffled the query arguments around java.lang.AssertionError: Expected: is "http: //sandbox.hortonworks.com/solr/KnoxSolrIntegration/select?q=*.*&indent= true &wt=json" but: was "http: //sandbox.hortonworks.com/solr/KnoxSolrIntegration/select?indent= true &q=*.*&wt=json" Maybe break down the 'outputUri' into host/path/query and check - there might be other better ways to do this. 2. Parser.parse( ) is deprecated so might be good to use Parser.parseTemplate() instead so that the test is not broken when the Parser.parse( ) api goes away. 3. Solr latest release is 6.3.0 any reason why the version is 5.5.0. I know it does not matter as long as the api remains same, but it might give an impressions that the api is not tested (which, I did and works great !) and updated for 6.3.0. Again, thanks !
          Hide
          smore Sandeep More added a comment -

          John McParland Sorry for being late to the party ! I'll take a look at the setup and report back, again much thanks !

          Show
          smore Sandeep More added a comment - John McParland Sorry for being late to the party ! I'll take a look at the setup and report back, again much thanks !
          Hide
          mcparlandjcgi John McParland added a comment -

          Thanks Kevin Risden

          The deployment could be a lot easier (for starters, containerized) so glad to hear any suggestions.

          You also raise a question that's been bugging me around working on Knox - how do we do integration testing?

          In a way, it shows the success of the project - one often only requires to create service.xml and rewrite.xml, but how do we verify what's in those files? And how do we validate that they shall work against a real Solr/other system behind the gateway?

          Show
          mcparlandjcgi John McParland added a comment - Thanks Kevin Risden The deployment could be a lot easier (for starters, containerized) so glad to hear any suggestions. You also raise a question that's been bugging me around working on Knox - how do we do integration testing? In a way, it shows the success of the project - one often only requires to create service.xml and rewrite.xml, but how do we verify what's in those files? And how do we validate that they shall work against a real Solr/other system behind the gateway?
          Hide
          risdenk Kevin Risden added a comment -

          Larry McCay - Are there any existing integration tests? If so I could help put together a Solr minicluster that runs as a JUnit test. I've done that for a few other projects.

          John McParland - The docs looks good. I have a few suggestions that would make the deployment potentially easier. I'll open issues/PRs against the github project you linked just in case it helps.

          Show
          risdenk Kevin Risden added a comment - Larry McCay - Are there any existing integration tests? If so I could help put together a Solr minicluster that runs as a JUnit test. I've done that for a few other projects. John McParland - The docs looks good. I have a few suggestions that would make the deployment potentially easier. I'll open issues/PRs against the github project you linked just in case it helps.
          Hide
          lmccay Larry McCay added a comment -

          Awesome!
          That will help and we should probably try and get a knox blog "article" out of that!

          Show
          lmccay Larry McCay added a comment - Awesome! That will help and we should probably try and get a knox blog "article" out of that!
          Hide
          mcparlandjcgi John McParland added a comment - - edited

          Thanks Larry McCay

          For some help with the setup, I've been placing some documentation on what I'm doing with Knox on GitHub.
          Here's an "article" on installing and configuring Solr in case it helps:

          Show
          mcparlandjcgi John McParland added a comment - - edited Thanks Larry McCay For some help with the setup, I've been placing some documentation on what I'm doing with Knox on GitHub. Here's an "article" on installing and configuring Solr in case it helps:
          Hide
          lmccay Larry McCay added a comment -

          No problem, John McParland - thanks for the follow up - it is much appreciated!
          It is definitely on the list to get reviewed and tested in the next couple days.

          This just happens to be one that requires additional setup to have the solr deployment available in order to test.

          Show
          lmccay Larry McCay added a comment - No problem, John McParland - thanks for the follow up - it is much appreciated! It is definitely on the list to get reviewed and tested in the next couple days. This just happens to be one that requires additional setup to have the solr deployment available in order to test.
          Hide
          mcparlandjcgi John McParland added a comment -

          Hi Larry McCay would it be possible to review this again with a view to including in the 0.11.0 release?

          Apologies for asking - I know it's getting busy.

          Show
          mcparlandjcgi John McParland added a comment - Hi Larry McCay would it be possible to review this again with a view to including in the 0.11.0 release? Apologies for asking - I know it's getting busy.
          Hide
          mcparlandjcgi John McParland added a comment -

          Thanks again Larry McCay for reviewing and explaining. I've uploaded a new patch, with commits squashed, which omits the authentication policy.

          I gave this a manual test too - providing credentials let's a Solr request go through Knox - failure to provide credentials results in a 401, so success on that front.

          Show
          mcparlandjcgi John McParland added a comment - Thanks again Larry McCay for reviewing and explaining. I've uploaded a new patch, with commits squashed, which omits the authentication policy. I gave this a manual test too - providing credentials let's a Solr request go through Knox - failure to provide credentials results in a 401, so success on that front.
          Hide
          mcparlandjcgi John McParland added a comment -

          Basic authentication now used to authenticate at Knox before passing onto Solr.

          Show
          mcparlandjcgi John McParland added a comment - Basic authentication now used to authenticate at Knox before passing onto Solr.
          Hide
          lmccay Larry McCay added a comment -

          Yes, John McParland - we want to NOT specify a particular provider name.
          That mechanism is for forcing the provider for a given service definition to always be that particular one.

          Anonymous means that Knox will allow the request through without authentication (anonymously) however the backend service may handle authentication itself. This is what is done by applications like Ambari, Ranger, etc.

          Hadoop provider allows for SPNEGO/kerberos and delegation token authentication.

          Shiro will be the most common but you don't want to limit it.
          Therefore, just specific authentication with no name.

          Show
          lmccay Larry McCay added a comment - Yes, John McParland - we want to NOT specify a particular provider name. That mechanism is for forcing the provider for a given service definition to always be that particular one. Anonymous means that Knox will allow the request through without authentication (anonymously) however the backend service may handle authentication itself. This is what is done by applications like Ambari, Ranger, etc. Hadoop provider allows for SPNEGO/kerberos and delegation token authentication. Shiro will be the most common but you don't want to limit it. Therefore, just specific authentication with no name.
          Hide
          mcparlandjcgi John McParland added a comment -

          Hi Larry McCay

          thanks for reviewing.

          The Solr API doesn't support authentication at all - therefore I want Knox to do the authentication itself using Basic credentials.
          On the opposite case, I don't want requests to go to Solr unless authentication by Knox.

          I looked at what the different options were for authentication and found three;

          • Anonymous
          • Hadoop
          • Shiro

          I though Anonymous would mean no additional Authentication after Knox did the basic credential check, and I'll confess I wan't sure what Hadoop was (Shiro seems obvious - delegate authentication to Shiro).

          So is the right approach in this case not to specify the "authentication" policy?

          Show
          mcparlandjcgi John McParland added a comment - Hi Larry McCay thanks for reviewing. The Solr API doesn't support authentication at all - therefore I want Knox to do the authentication itself using Basic credentials. On the opposite case, I don't want requests to go to Solr unless authentication by Knox. I looked at what the different options were for authentication and found three; Anonymous Hadoop Shiro I though Anonymous would mean no additional Authentication after Knox did the basic credential check, and I'll confess I wan't sure what Hadoop was (Shiro seems obvious - delegate authentication to Shiro). So is the right approach in this case not to specify the "authentication" policy?
          Hide
          lmccay Larry McCay added a comment -

          Hi John McParland -

          Regarding the service definition decisions in the following:

          +<service role="SOLRAPI" name="solr" version="5.5.0">
          +    <policies>
          +        <policy role="webappsec"/>
          +        <policy role="authentication" name="Anonymous"/>
          +        <policy role="rewrite"/>
          +        <policy role="authorization"/>
          +    </policies>
          +    <routes>
          +        <route path="/solr/**/**?**">
          +             <rewrite apply="SOLRAPI/solr/inbound/query" to="request.url"/>
          +        </route>
          +    </routes>
          +     <dispatch classname="org.apache.hadoop.gateway.dispatch.PassAllHeadersDispatch"/>
          +</service>
          

          can you describe why we want to require that all authentication to the solr API be delegated to solr itself?

          I notice that you pass basic credentials to Knox in the evidence file but not to Solr when going direct.
          I suspect that you expect Knox to authentication using the basic credentials in that case.
          When you actually specify the name "Anonymous" as the authentication provider in the above - you are indicating that Knox will allow Anonymous access to the service. If the service itself handles its own authentication then we will allow that and pass along any cookies that are set as appropriate.

          That is actually what the dispatch that you defined for solr above does.
          <dispatch classname="org.apache.hadoop.gateway.dispatch.PassAllHeadersDispatch"/>

          This may be appropriate for solr, I haven't had a chance to dig into solr service details yet.
          Many services in the Hadoop ecosystem support the notion of Trusted Proxies.
          This would allow us to authenticate a request intended for Solr at the gateway then during dispatch we would authenticate as the trusted proxy (in this case as knox) and pass a doas for the authenticated user.

          Does solr support trusted proxies in this way?

          Show
          lmccay Larry McCay added a comment - Hi John McParland - Regarding the service definition decisions in the following: +<service role="SOLRAPI" name="solr" version="5.5.0"> + <policies> + <policy role="webappsec"/> + <policy role="authentication" name="Anonymous"/> + <policy role="rewrite"/> + <policy role="authorization"/> + </policies> + <routes> + <route path="/solr/**/**?**"> + <rewrite apply="SOLRAPI/solr/inbound/query" to="request.url"/> + </route> + </routes> + <dispatch classname="org.apache.hadoop.gateway.dispatch.PassAllHeadersDispatch"/> +</service> can you describe why we want to require that all authentication to the solr API be delegated to solr itself? I notice that you pass basic credentials to Knox in the evidence file but not to Solr when going direct. I suspect that you expect Knox to authentication using the basic credentials in that case. When you actually specify the name "Anonymous" as the authentication provider in the above - you are indicating that Knox will allow Anonymous access to the service. If the service itself handles its own authentication then we will allow that and pass along any cookies that are set as appropriate. That is actually what the dispatch that you defined for solr above does. <dispatch classname="org.apache.hadoop.gateway.dispatch.PassAllHeadersDispatch"/> This may be appropriate for solr, I haven't had a chance to dig into solr service details yet. Many services in the Hadoop ecosystem support the notion of Trusted Proxies. This would allow us to authenticate a request intended for Solr at the gateway then during dispatch we would authenticate as the trusted proxy (in this case as knox) and pass a doas for the authenticated user. Does solr support trusted proxies in this way?
          Hide
          lmccay Larry McCay added a comment -

          John McParland - Thank you for squashing the commits!
          That is much easier to review.
          Let me take a look at that today.

          Sorry for the delay on this.
          I will set the status to patch available.

          Show
          lmccay Larry McCay added a comment - John McParland - Thank you for squashing the commits! That is much easier to review. Let me take a look at that today. Sorry for the delay on this. I will set the status to patch available.
          Hide
          mcparlandjcgi John McParland added a comment -

          HI Larry McCayhave you had a chance to review the patch?

          Thanks,

          John

          Show
          mcparlandjcgi John McParland added a comment - HI Larry McCay have you had a chance to review the patch? Thanks, John
          Hide
          mcparlandjcgi John McParland added a comment - - edited

          Hi Larry McCay I've uploaded a new patch with the commits squashed.

          See KNOX-528_squashed.patch

          Show
          mcparlandjcgi John McParland added a comment - - edited Hi Larry McCay I've uploaded a new patch with the commits squashed. See KNOX-528_squashed.patch
          Hide
          lmccay Larry McCay added a comment -

          John McParland - could I bother you to squash the commits in the patch into a single commit for this JIRA?
          That makes it much easier to review and can reflect the history of the change within the project without all your dev commits.

          Show
          lmccay Larry McCay added a comment - John McParland - could I bother you to squash the commits in the patch into a single commit for this JIRA? That makes it much easier to review and can reflect the history of the change within the project without all your dev commits.
          Hide
          lmccay Larry McCay added a comment -

          Great to see a patch ready for this, John McParland!
          I will try and carve out some time to review it today or over the next couple of days.

          Your summary comment will be very helpful in the review.

          Show
          lmccay Larry McCay added a comment - Great to see a patch ready for this, John McParland ! I will try and carve out some time to review it today or over the next couple of days. Your summary comment will be very helpful in the review.
          Hide
          mcparlandjcgi John McParland added a comment -

          Corrected the patch - was missing a commit.

          Show
          mcparlandjcgi John McParland added a comment - Corrected the patch - was missing a commit.
          Hide
          mcparlandjcgi John McParland added a comment -
          • knoxSolrTestEvidence.txt
            • The response generated when the API call was made through Knox
          • solrTestEvidence.txt
            • The response generated when the API call was made directly through Solr.
          Show
          mcparlandjcgi John McParland added a comment - knoxSolrTestEvidence.txt The response generated when the API call was made through Knox solrTestEvidence.txt The response generated when the API call was made directly through Solr.
          Hide
          mcparlandjcgi John McParland added a comment - - edited

          Added service and rewrite definitions for Solr support via Knox.

          Key decisions

          • used a Solr service definition obtained from the topology (as opposed to Zookeeper): this is in line with proposed setup for client where there will only be one Solr
          • Installed Solr on HDP 2.4 using the HDP Search product
          • Only proxied the API via Knox - no need nor desire to proxy the UI

          Patch includes

          • service.xml definition for Solr with route for the API
          • rewrite.xml definition for the re-write rules
          • update to UrlRewriteProcessorTest that includes a JUnit test for the Solr re-write rule.

          Example topology definition for Solr (based on Hortonworks Data Platform 2.4 sandbox)

              <service>
                  <role>SOLRAPI</role>
                  <url>http://sandbox.hortonworks.com:8983/solr</url>
              </service>
          

          Also attached evidence of running it - first an example of output from Solr directory, the second of the same query run through Knox. Only difference is the URLs and timing!

          I would like to acknowledge Sandeep More's help in getting the pattern matching right and pointers in debugging. Thanks!

          Show
          mcparlandjcgi John McParland added a comment - - edited Added service and rewrite definitions for Solr support via Knox. Key decisions used a Solr service definition obtained from the topology (as opposed to Zookeeper): this is in line with proposed setup for client where there will only be one Solr Installed Solr on HDP 2.4 using the HDP Search product Only proxied the API via Knox - no need nor desire to proxy the UI Patch includes service.xml definition for Solr with route for the API rewrite.xml definition for the re-write rules update to UrlRewriteProcessorTest that includes a JUnit test for the Solr re-write rule. Example topology definition for Solr (based on Hortonworks Data Platform 2.4 sandbox) <service> <role>SOLRAPI</role> <url>http://sandbox.hortonworks.com:8983/solr</url> </service> Also attached evidence of running it - first an example of output from Solr directory, the second of the same query run through Knox. Only difference is the URLs and timing! I would like to acknowledge Sandeep More 's help in getting the pattern matching right and pointers in debugging. Thanks!
          Hide
          kristopherkane Kristopher Kane added a comment -

          I don't have much to add other than it is exciting to have community
          additions like this. The need for Solr fell off my radar but happy to see
          your interest, Kevin, as I've been following on the Solr lists.

          Kris

          Show
          kristopherkane Kristopher Kane added a comment - I don't have much to add other than it is exciting to have community additions like this. The need for Solr fell off my radar but happy to see your interest, Kevin, as I've been following on the Solr lists. Kris
          Hide
          lmccay Larry McCay added a comment -

          Kevin Risden - thanks for the response.

          Is it possible to have one service? There really isn't a difference between the Solr UI and the APIs. It would be awesome to be able to visit a single Knox endpoint for both. Below expands a bit more on the relationship between the UI and the APIs and maybe that helps.

          The problem with coupling them together is that you lose the ability to protect the REST API independently from the UI. For example, you currently have the UI using the anonymous provider which means Knox is delegating any authentication to the UI itself.

          If you leave the API definition in there as well then you won't be able to expose the API independently with, say, HTTP basic auth with authentication against LDAP. This is the most common provider use for APIs but you can also put any number of other ones in there as well for a HTTP header based preauthentication, etc.

          Show
          lmccay Larry McCay added a comment - Kevin Risden - thanks for the response. Is it possible to have one service? There really isn't a difference between the Solr UI and the APIs. It would be awesome to be able to visit a single Knox endpoint for both. Below expands a bit more on the relationship between the UI and the APIs and maybe that helps. The problem with coupling them together is that you lose the ability to protect the REST API independently from the UI. For example, you currently have the UI using the anonymous provider which means Knox is delegating any authentication to the UI itself. If you leave the API definition in there as well then you won't be able to expose the API independently with, say, HTTP basic auth with authentication against LDAP. This is the most common provider use for APIs but you can also put any number of other ones in there as well for a HTTP header based preauthentication, etc.
          Hide
          risdenk Kevin Risden added a comment -

          The following is where most of the service tests go that need to test expectations of the rewrite rules and dispatch classes within the definition for a given service: https://github.com/apache/knox/blob/master/gateway-test/src/test/java/org/apache/hadoop/gateway/GatewayBasicFuncTest.java
          This is especially important for when there are nuances to what rewrite or dispatch need to do.

          Ah ok I'll take a look.

          That said, there are still a few things about he POC that needs to be addressed:
          1. It seems that the solr definition provided in your POC is really only for the UI. So, there needs to be a corresponding REST API definition as well. This is generally done with a SOLRUI and SOLR service pair.

          Is it possible to have one service? There really isn't a difference between the Solr UI and the APIs. It would be awesome to be able to visit a single Knox endpoint for both. Below expands a bit more on the relationship between the UI and the APIs and maybe that helps.

          2. The current definition is using the Anonymous authentication provider. This is commonly used for UIs that provide their own authentication with form-based login, HTTP basic auth or even with KnoxSSO integration, etc. It doesn't look like solr UI is doing any authentication for the user. Unauthenticated UI access should be avoided. We should discuss what is needed there further.

          Ah wasn't really sure what that was for other than just copied from the proxying UI page. Right now with the basic setup there is no UI authentication. Solr really doesn't support UI authentication per say. The authentication is actually at the API layer itself. The UI really just needs rewriting. The APIs have support for basic and SPNEGO Kerberos authentication if you enable it in Solr (not the easiest to do right now, but possible).

          3. The dispatch being used is another common one for UIs and won't likely be sufficient for REST APIs though we can discuss that as well. It depends on what straight API access requires as far as credentials or identity propagation.

          Just to provide a little more background on the Solr UI, it really just calls out to the Solr REST APIs and otherwise is just an angular UI.

          4. I need some clarity on the topology service descriptor. The URL within the default.xml topology for the service seems strange (I probably just need it explained to me) and there doesn't seem to be any HA specific URLs. I expected to see multiple URLs or Zookeeper specifics for determining the URLs, etc.

          Right now for the quick demo there is only one Solr host. Each host in a Solr cluster knows how to forward requests to the right host. This means that Knox just needs to be able to pick one available host. I have tested this a little by just adding another url and Knox "does the right thing". I am using the Knox built in HA provider to do failover if it determines the first URL is down.

          The Solr url format is as follows:

          http://SOLR_HOSTNAME:SOLR_PORT/solr

          The /solr is hardcoded with Solr 5.x+ since it is meant to be run as is. The docker example has a format that looks a bit weird since the name I gave the Solr container is "solr".

          Show
          risdenk Kevin Risden added a comment - The following is where most of the service tests go that need to test expectations of the rewrite rules and dispatch classes within the definition for a given service: https://github.com/apache/knox/blob/master/gateway-test/src/test/java/org/apache/hadoop/gateway/GatewayBasicFuncTest.java This is especially important for when there are nuances to what rewrite or dispatch need to do. Ah ok I'll take a look. That said, there are still a few things about he POC that needs to be addressed: 1. It seems that the solr definition provided in your POC is really only for the UI. So, there needs to be a corresponding REST API definition as well. This is generally done with a SOLRUI and SOLR service pair. Is it possible to have one service? There really isn't a difference between the Solr UI and the APIs. It would be awesome to be able to visit a single Knox endpoint for both. Below expands a bit more on the relationship between the UI and the APIs and maybe that helps. 2. The current definition is using the Anonymous authentication provider. This is commonly used for UIs that provide their own authentication with form-based login, HTTP basic auth or even with KnoxSSO integration, etc. It doesn't look like solr UI is doing any authentication for the user. Unauthenticated UI access should be avoided. We should discuss what is needed there further. Ah wasn't really sure what that was for other than just copied from the proxying UI page. Right now with the basic setup there is no UI authentication. Solr really doesn't support UI authentication per say. The authentication is actually at the API layer itself. The UI really just needs rewriting. The APIs have support for basic and SPNEGO Kerberos authentication if you enable it in Solr (not the easiest to do right now, but possible). 3. The dispatch being used is another common one for UIs and won't likely be sufficient for REST APIs though we can discuss that as well. It depends on what straight API access requires as far as credentials or identity propagation. Just to provide a little more background on the Solr UI, it really just calls out to the Solr REST APIs and otherwise is just an angular UI. 4. I need some clarity on the topology service descriptor. The URL within the default.xml topology for the service seems strange (I probably just need it explained to me) and there doesn't seem to be any HA specific URLs. I expected to see multiple URLs or Zookeeper specifics for determining the URLs, etc. Right now for the quick demo there is only one Solr host. Each host in a Solr cluster knows how to forward requests to the right host. This means that Knox just needs to be able to pick one available host. I have tested this a little by just adding another url and Knox "does the right thing". I am using the Knox built in HA provider to do failover if it determines the first URL is down. The Solr url format is as follows: http://SOLR_HOSTNAME:SOLR_PORT/solr The /solr is hardcoded with Solr 5.x+ since it is meant to be run as is. The docker example has a format that looks a bit weird since the name I gave the Solr container is "solr".
          Hide
          lmccay Larry McCay added a comment -

          Hi Kevin Risden - The following is where most of the service tests go that need to test expectations of the rewrite rules and dispatch classes within the definition for a given service: https://github.com/apache/knox/blob/master/gateway-test/src/test/java/org/apache/hadoop/gateway/GatewayBasicFuncTest.java

          This is especially important for when there are nuances to what rewrite or dispatch need to do.

          That said, there are still a few things about he POC that needs to be addressed:

          1. It seems that the solr definition provided in your POC is really only for the UI. So, there needs to be a corresponding REST API definition as well. This is generally done with a SOLRUI and SOLR service pair.
          2. The current definition is using the Anonymous authentication provider. This is commonly used for UIs that provide their own authentication with form-based login, HTTP basic auth or even with KnoxSSO integration, etc. It doesn't look like solr UI is doing any authentication for the user. Unauthenticated UI access should be avoided. We should discuss what is needed there further.
          3. The dispatch being used is another common one for UIs and won't likely be sufficient for REST APIs though we can discuss that as well. It depends on what straight API access requires as far as credentials or identity propagation.
          4. I need some clarity on the topology service descriptor. The URL within the default.xml topology for the service seems strange (I probably just need it explained to me) and there doesn't seem to be any HA specific URLs. I expected to see multiple URLs or Zookeeper specifics for determining the URLs, etc.

          Thanks for this work - it will be very useful for the community!

          Show
          lmccay Larry McCay added a comment - Hi Kevin Risden - The following is where most of the service tests go that need to test expectations of the rewrite rules and dispatch classes within the definition for a given service: https://github.com/apache/knox/blob/master/gateway-test/src/test/java/org/apache/hadoop/gateway/GatewayBasicFuncTest.java This is especially important for when there are nuances to what rewrite or dispatch need to do. That said, there are still a few things about he POC that needs to be addressed: 1. It seems that the solr definition provided in your POC is really only for the UI. So, there needs to be a corresponding REST API definition as well. This is generally done with a SOLRUI and SOLR service pair. 2. The current definition is using the Anonymous authentication provider. This is commonly used for UIs that provide their own authentication with form-based login, HTTP basic auth or even with KnoxSSO integration, etc. It doesn't look like solr UI is doing any authentication for the user. Unauthenticated UI access should be avoided. We should discuss what is needed there further. 3. The dispatch being used is another common one for UIs and won't likely be sufficient for REST APIs though we can discuss that as well. It depends on what straight API access requires as far as credentials or identity propagation. 4. I need some clarity on the topology service descriptor. The URL within the default.xml topology for the service seems strange (I probably just need it explained to me) and there doesn't seem to be any HA specific URLs. I expected to see multiple URLs or Zookeeper specifics for determining the URLs, etc. Thanks for this work - it will be very useful for the community!
          Hide
          risdenk Kevin Risden added a comment -

          Larry McCay - Can you point me to an example of existing tests?

          Show
          risdenk Kevin Risden added a comment - Larry McCay - Can you point me to an example of existing tests?
          Hide
          lmccay Larry McCay added a comment -

          Ahhh - I missed that somehow.
          Thanks!

          Show
          lmccay Larry McCay added a comment - Ahhh - I missed that somehow. Thanks!
          Hide
          risdenk Kevin Risden added a comment -

          The docker-compose.yml file describes the environment. I have it built a Knox image since one doesn't exist. For Solr, I am using the one published on Docker Hub already. The image tag defines it as Solr: alpine. This is how Solr is getting started.

          Show
          risdenk Kevin Risden added a comment - The docker-compose.yml file describes the environment. I have it built a Knox image since one doesn't exist. For Solr, I am using the one published on Docker Hub already. The image tag defines it as Solr: alpine. This is how Solr is getting started.
          Hide
          lmccay Larry McCay added a comment -

          I patch would be great.
          I think that we will also need tests for the SOLR REST API or at least a sample that can be tested at apache release time.

          Question about the docker environment here...
          How is solr getting installed and started - don't see it in any of the files in the github project.
          Is it in the base image of the machine already?

          Show
          lmccay Larry McCay added a comment - I patch would be great. I think that we will also need tests for the SOLR REST API or at least a sample that can be tested at apache release time. Question about the docker environment here... How is solr getting installed and started - don't see it in any of the files in the github project. Is it in the base image of the machine already?
          Hide
          risdenk Kevin Risden added a comment -

          1. Hmmm OK I'll have to look into the old ui link. The new ui is the only one actively being improved so might not be a big deal.
          2. Adding a collection requires a config set to be previously added. I've been meaning to file a Solr Jira with that issue since the error message is highly misleading. I can have a config set added so creating a collection should be possible.

          Thanks for looking at the example. Definitely let me know if you find anything else. I can take a service and create a patch out of it too if that would help.

          Show
          risdenk Kevin Risden added a comment - 1. Hmmm OK I'll have to look into the old ui link. The new ui is the only one actively being improved so might not be a big deal. 2. Adding a collection requires a config set to be previously added. I've been meaning to file a Solr Jira with that issue since the error message is highly misleading. I can have a config set added so creating a collection should be possible. Thanks for looking at the example. Definitely let me know if you find anything else. I can take a service and create a patch out of it too if that would help.
          Hide
          lmccay Larry McCay added a comment -

          Kevin Risden - this is cool.
          I like the docker approach to providing this POC!

          There do still seem to be some rewrite issues in here.
          1. The links to switch between old and new UIs need to be rewritten yet.
          2. It seems that adding a collection doesn't seem to work - unless I just can't figure out what it needs to be given. The provided path can't end with a slash - even though I'm not putting a slash in it....

          The REST APIs do seem to be working.

          Show
          lmccay Larry McCay added a comment - Kevin Risden - this is cool. I like the docker approach to providing this POC! There do still seem to be some rewrite issues in here. 1. The links to switch between old and new UIs need to be rewritten yet. 2. It seems that adding a collection doesn't seem to work - unless I just can't figure out what it needs to be given. The provided path can't end with a slash - even though I'm not putting a slash in it.... The REST APIs do seem to be working.
          Hide
          risdenk Kevin Risden added a comment -

          Just a note I was able to get some basic rewrite rules for the Solr admin UI working. No more major errors when proxying through Knox. I would be curious if others could try it out.

          Show
          risdenk Kevin Risden added a comment - Just a note I was able to get some basic rewrite rules for the Solr admin UI working. No more major errors when proxying through Knox. I would be curious if others could try it out.
          Hide
          risdenk Kevin Risden added a comment -

          I've been playing with Knox proxying Solr on and off for the last two months. Since Knox 0.7.x+ has support for HA and that allows specifying multiple URLs, this means you can easily proxy Solr requests with Knox with a service. The solr folder in the repo has an example service. The only thing left is fixing the rewrite rules for the Solr admin UI (should be easier now that https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox has been created this week).

          https://github.com/risdenk/knox_solr_testing

          Show
          risdenk Kevin Risden added a comment - I've been playing with Knox proxying Solr on and off for the last two months. Since Knox 0.7.x+ has support for HA and that allows specifying multiple URLs, this means you can easily proxy Solr requests with Knox with a service. The solr folder in the repo has an example service. The only thing left is fixing the rewrite rules for the Solr admin UI (should be easier now that https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox has been created this week). https://github.com/risdenk/knox_solr_testing
          Hide
          lmccay Larry McCay added a comment -

          Great to know.
          Thanks, Rick!

          Show
          lmccay Larry McCay added a comment - Great to know. Thanks, Rick!
          Hide
          rkellogg Rick Kellogg added a comment - - edited

          In our scenario, the use of Solr Cloud is mandatory (with many thousands of cores). The Collections API is of secondary importance to us.

          Show
          rkellogg Rick Kellogg added a comment - - edited In our scenario, the use of Solr Cloud is mandatory (with many thousands of cores). The Collections API is of secondary importance to us.
          Hide
          lmccay Larry McCay added a comment -

          Hi Kristopher Kane and Rick Kellogg -

          Do either of you have a sense for whether initial support for Solr without Solr Cloud and/or collections would be useful for real world usecases?

          If we can add value in the near term with something simple that we can evolve over time then I think that would be worth considering.

          Show
          lmccay Larry McCay added a comment - Hi Kristopher Kane and Rick Kellogg - Do either of you have a sense for whether initial support for Solr without Solr Cloud and/or collections would be useful for real world usecases? If we can add value in the near term with something simple that we can evolve over time then I think that would be worth considering.
          Hide
          lmccay Larry McCay added a comment -

          Great to see this discussion happening here.
          It is important for ensuring that the implementation meets the requirements of actual customers.

          That said, I think it may be healthy to enumerate the immediate driving usecases and how they may or may not require the collections API and/or Solr Cloud.

          With those in place, we can then try and make sure that we have a path forward for future usecases.
          Thoughts?

          Show
          lmccay Larry McCay added a comment - Great to see this discussion happening here. It is important for ensuring that the implementation meets the requirements of actual customers. That said, I think it may be healthy to enumerate the immediate driving usecases and how they may or may not require the collections API and/or Solr Cloud. With those in place, we can then try and make sure that we have a path forward for future usecases. Thoughts?
          Hide
          rkellogg Rick Kellogg added a comment -

          In the short term, we are restricted to SOLR 4.10.2 as that is what is included in HortonWorks Search at this time. Longer term we will upgrade to SOLR 5.x which includes many enhancements plus increased maximum core capacity.

          The client within SOLR 4.x is HttpSolrServer and has been renamed to HttpSolrClient in 5.x. I think this would be the preferred client in both cases and would expect both to be compatible from a Knox standpoint. I might be wrong here.

          Just reviewed the SOLR Collections API (https://cwiki.apache.org/confluence/display/solr/Collections+API), I see the complications. Knox would turn into a sort of federation of clusters and therefore need to know which cores are located in specific clusters. Fairly tricky issue in terms of features ownership. We definitely do not want to replicate features that SOLR Cloud might address in the future. Will do a bit more research and provide input.

          Show
          rkellogg Rick Kellogg added a comment - In the short term, we are restricted to SOLR 4.10.2 as that is what is included in HortonWorks Search at this time. Longer term we will upgrade to SOLR 5.x which includes many enhancements plus increased maximum core capacity. The client within SOLR 4.x is HttpSolrServer and has been renamed to HttpSolrClient in 5.x. I think this would be the preferred client in both cases and would expect both to be compatible from a Knox standpoint. I might be wrong here. Just reviewed the SOLR Collections API ( https://cwiki.apache.org/confluence/display/solr/Collections+API ), I see the complications. Knox would turn into a sort of federation of clusters and therefore need to know which cores are located in specific clusters. Fairly tricky issue in terms of features ownership. We definitely do not want to replicate features that SOLR Cloud might address in the future. Will do a bit more research and provide input.
          Hide
          kristopherkane Kristopher Kane added a comment - - edited

          Which version of Solr did you intend on using this solution with? CommonsHttpSolrServer was removed but it seems HttpSolrClient will fit the same need. http://lucene.apache.org/solr/5_0_0/solr-solrj/org/apache/solr/client/solrj/impl/HttpSolrClient.html.

          Also, do you intend on using the the collections API?

          Kris

          Show
          kristopherkane Kristopher Kane added a comment - - edited Which version of Solr did you intend on using this solution with? CommonsHttpSolrServer was removed but it seems HttpSolrClient will fit the same need. http://lucene.apache.org/solr/5_0_0/solr-solrj/org/apache/solr/client/solrj/impl/HttpSolrClient.html . Also, do you intend on using the the collections API? Kris
          Hide
          rkellogg Rick Kellogg added a comment -

          Another option would be using preemptive authentication with CommonsHttpSolrServer similar to the following:

          String USERNAME = "x";
          String PASSWORD = "x";
          String HOSTNAME = "x";
          String ROUTEID = "x";
          CommonsHttpSolrServer server = new CommonsHttpSolrServer("http://" + USERNAME + ":" + PASSWORD + "@" + HOSTNAME + "/" + ROUTEID + "/solr/collection1");
          Credentials defaultcreds = new UsernamePasswordCredentials(USERNAME, PASSWORD);
          server.getHttpClient().getState().setCredentials(new AuthScope(HOSTNAME, 80, AuthScope.ANY_REALM), defaultcreds);
          server.getHttpClient().getParams().setAuthenticationPreemptive(true);

          This might be the best option and not require any changes to the SOLRJ baseline.

          Show
          rkellogg Rick Kellogg added a comment - Another option would be using preemptive authentication with CommonsHttpSolrServer similar to the following: String USERNAME = "x"; String PASSWORD = "x"; String HOSTNAME = "x"; String ROUTEID = "x"; CommonsHttpSolrServer server = new CommonsHttpSolrServer("http://" + USERNAME + ":" + PASSWORD + "@" + HOSTNAME + "/" + ROUTEID + "/solr/collection1"); Credentials defaultcreds = new UsernamePasswordCredentials(USERNAME, PASSWORD); server.getHttpClient().getState().setCredentials(new AuthScope(HOSTNAME, 80, AuthScope.ANY_REALM), defaultcreds); server.getHttpClient().getParams().setAuthenticationPreemptive(true); This might be the best option and not require any changes to the SOLRJ baseline.
          Hide
          rkellogg Rick Kellogg added a comment - - edited

          Upon review of the SOLRJ javadoc, it appears that we might need to implement our own SolrClient similar to the CloudSolrClient which uses the Knox endpoint information instead of the Zookeeper settings.

          Show
          rkellogg Rick Kellogg added a comment - - edited Upon review of the SOLRJ javadoc, it appears that we might need to implement our own SolrClient similar to the CloudSolrClient which uses the Knox endpoint information instead of the Zookeeper settings.
          Hide
          rkellogg Rick Kellogg added a comment -

          Good questions.

          Optimally we would like to continue to use the standard SOLRJ client (both binary and XML formats). Not certain this is possible. Apache Knox would just operate like an invisible proxy that hides the physical cluster configuration and provides security and logging.

          Another goal would be to allow federation of multiple SOLR clusters transparently behind a single endpoint. Assume cores are distinct between clusters. In our deployment we will have tens of thousands of cores in just a few years. We do not necessarily need fine grained security at the core level but at least at the macro level. Suggest we have a pluggable security module here so as to allow security at the SOLR cluster level or finer degree of control. Will give security a bit more thought and try to contribute where I can.

          Show
          rkellogg Rick Kellogg added a comment - Good questions. Optimally we would like to continue to use the standard SOLRJ client (both binary and XML formats). Not certain this is possible. Apache Knox would just operate like an invisible proxy that hides the physical cluster configuration and provides security and logging. Another goal would be to allow federation of multiple SOLR clusters transparently behind a single endpoint. Assume cores are distinct between clusters. In our deployment we will have tens of thousands of cores in just a few years. We do not necessarily need fine grained security at the core level but at least at the macro level. Suggest we have a pluggable security module here so as to allow security at the SOLR cluster level or finer degree of control. Will give security a bit more thought and try to contribute where I can.
          Hide
          kristopherkane Kristopher Kane added a comment -

          Hi Rick Kellogg . Any specific needs? I assume SolrCloud and am thinking that a Knox role would represent a SolrCloud collection - or possibly - configure multiple collections per the SolrCloud role in the topology. This will allow Knox to keep warm a list of active servers out of ZK. Another approach is to query ZK per API call by parsing the collection out of the URL. Knox could then remember these collections and keep the list warm for as long as the Knox server is running.

          Show
          kristopherkane Kristopher Kane added a comment - Hi Rick Kellogg . Any specific needs? I assume SolrCloud and am thinking that a Knox role would represent a SolrCloud collection - or possibly - configure multiple collections per the SolrCloud role in the topology. This will allow Knox to keep warm a list of active servers out of ZK. Another approach is to query ZK per API call by parsing the collection out of the URL. Knox could then remember these collections and keep the list warm for as long as the Knox server is running.
          Hide
          lmccay Larry McCay added a comment -

          Hi Rick Kellogg -

          Thank you for expressing interest in accessing Solr through Knox!
          Kristopher Kane was interested in this work and I believe has done some proof of concept work on it.

          Are you looking to provide a patch for this feature?

          If so, perhaps you can sync and/or collaborate on it and share your usecases and insights into implementation plans.
          Feel free to discuss here on the jira or on the dev@ list.

          Thanks again!

          Show
          lmccay Larry McCay added a comment - Hi Rick Kellogg - Thank you for expressing interest in accessing Solr through Knox! Kristopher Kane was interested in this work and I believe has done some proof of concept work on it. Are you looking to provide a patch for this feature? If so, perhaps you can sync and/or collaborate on it and share your usecases and insights into implementation plans. Feel free to discuss here on the jira or on the dev@ list. Thanks again!

            People

            • Assignee:
              mcparlandjcgi John McParland
              Reporter:
              rkellogg Rick Kellogg
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development