Solr
  1. Solr
  2. SOLR-6833

bin/solr -e foo should not use server/solr as the SOLR_HOME

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0
    • Component/s: None
    • Labels:
      None

      Description

      i think it's weird right now that running bin/solr with the "-e" (example) option causes it to create example solr instances inside the server directory.

      i think that's fine for running solr "normally" (ie: "start") but if you use "-e" that seems like the solr.solr.home for those example should instead be created under $SOLR_TIP/example.

      I would even go so far as to suggest that the log files created should live in that directory as well.

      1. SOLR-6833.patch
        8 kB
        Timothy Potter

        Issue Links

          Activity

          Hide
          Timothy Potter added a comment -

          Alexandre Rafalovitch suggested this too ...

          Couple of things:

          1. Should the distribution bundle include example/solr/solr.xml? Or should we just copy server/solr/solr.xml to example/solr in the script if it's not there already? I guess I prefer copying it over vs. maintaining another file in version control.
          1. I like the idea of putting the example logs under example/ too. Currently, the server/resources/log4j.properties file just uses the cwd of the Jetty server to determine the location of the log file, specifically, it does:
          solr.log=logs/
          ...
          log4j.appender.file.File=${solr.log}/solr.log
          

          This translates to server/logs at runtime. What I'd like to do is change server/resources/log4j.properties to:

          solr.log=${solr.solr.home}/logs/
          ...
          log4j.appender.file.File=${solr.log}/solr.log
          

          However, this will put logs under server/solr/logs instead of server/logs, which I actually like better since it allows you to reuse the server directory for multiple Solr nodes running on the same server with different solr.solr.homes, i.e. the following commands will result in two Solr nodes running on the same server, sharing the server directory but having different solr.solr.home directories:

          bin/solr -p 8983 -s node1
          bin/solr -p 8984 -s node2
          

          If you do this now, then the log files will conflict with each other, so putting the logs directory under solr.solr.home will work better. If we don't think reusing the server directory is a big deal and want to keep server/logs, then I can also do: solr.log=$

          {solr.solr.home}

          /../logs/ which will keep logs in server/logs or example/logs. Sorry if this is a bit pedantic but I want to make sure we're all in agreement with these changes since we're getting close to the 5x release.

          Show
          Timothy Potter added a comment - Alexandre Rafalovitch suggested this too ... Couple of things: Should the distribution bundle include example/solr/solr.xml ? Or should we just copy server/solr/solr.xml to example/solr in the script if it's not there already? I guess I prefer copying it over vs. maintaining another file in version control. I like the idea of putting the example logs under example/ too. Currently, the server/resources/log4j.properties file just uses the cwd of the Jetty server to determine the location of the log file, specifically, it does: solr.log=logs/ ... log4j.appender.file.File=${solr.log}/solr.log This translates to server/logs at runtime. What I'd like to do is change server/resources/log4j.properties to: solr.log=${solr.solr.home}/logs/ ... log4j.appender.file.File=${solr.log}/solr.log However, this will put logs under server/solr/logs instead of server/logs , which I actually like better since it allows you to reuse the server directory for multiple Solr nodes running on the same server with different solr.solr.homes, i.e. the following commands will result in two Solr nodes running on the same server, sharing the server directory but having different solr.solr.home directories: bin/solr -p 8983 -s node1 bin/solr -p 8984 -s node2 If you do this now, then the log files will conflict with each other, so putting the logs directory under solr.solr.home will work better. If we don't think reusing the server directory is a big deal and want to keep server/logs, then I can also do: solr.log=$ {solr.solr.home} /../logs/ which will keep logs in server/logs or example/logs. Sorry if this is a bit pedantic but I want to make sure we're all in agreement with these changes since we're getting close to the 5x release.
          Hide
          Anshum Gupta added a comment - - edited

          +1 on creating the instance directory in $SOLR_TIP/example.

          Also, it makes sense to have the log directory in solr.solr.home. I personally would like it to be directly under solr.solr.home/logs instead of an added level. Makes things easier to cleanup too.

          Show
          Anshum Gupta added a comment - - edited +1 on creating the instance directory in $SOLR_TIP/example. Also, it makes sense to have the log directory in solr.solr.home. I personally would like it to be directly under solr.solr.home/logs instead of an added level. Makes things easier to cleanup too.
          Hide
          Hoss Man added a comment -

          Should the distribution bundle include example/solr/solr.xml? Or should we just copy server/solr/solr.xml to example/solr in the script if it's not there already?

          i think copying is the right way to go ... as far as the specific path...

          i think a big question is if you intend/expect the different examples to co-mingle. If i run "bin/solr -e techproducts" and then later i run "bin/solr -e schemaless" is the expectation that i now have a single instance of solr with both the techproducts and schemaless cores up and running in it? what about example-DIH, which isn't a configset and already has it's own dir in examples?

          what about the cloud examples?

          i think each "-e foo" option should create it's own "example/foo" directory (if it doesn't already exist - DIH always will) and create a stub solr.xml there. in the case of "-e cloud" it has to go one more level for each server: "example/cloud/nodeN"

          but the bottom line mantra should be: if it's created as part of an example, it lives in example/

          Show
          Hoss Man added a comment - Should the distribution bundle include example/solr/solr.xml? Or should we just copy server/solr/solr.xml to example/solr in the script if it's not there already? i think copying is the right way to go ... as far as the specific path... i think a big question is if you intend/expect the different examples to co-mingle. If i run "bin/solr -e techproducts" and then later i run "bin/solr -e schemaless" is the expectation that i now have a single instance of solr with both the techproducts and schemaless cores up and running in it? what about example-DIH, which isn't a configset and already has it's own dir in examples? what about the cloud examples? i think each "-e foo" option should create it's own "example/foo" directory (if it doesn't already exist - DIH always will) and create a stub solr.xml there. in the case of "-e cloud" it has to go one more level for each server: "example/cloud/nodeN" but the bottom line mantra should be: if it's created as part of an example, it lives in example/
          Hide
          Alexandre Rafalovitch added a comment -

          what about the cloud examples?

          The cloud examples at the moment copy the whole server directory into node1 and node2 in Solr root and then run from those. And re-running them requires figuring out whatever command line the script used to start them and doing it manually. There is no bin/start "previous cloud example"

          I am quite unhappy about that, especially with the rest of the stuff discussed above showing up in the server directory. I was told the latest code will carefully avoid extra directories, but I am not totally sold.

          Also, the node1/node2 is nearly invisible in the directory and will cause confusion. I think they should be at least prefixed with the example name or something.

          Show
          Alexandre Rafalovitch added a comment - what about the cloud examples? The cloud examples at the moment copy the whole server directory into node1 and node2 in Solr root and then run from those. And re-running them requires figuring out whatever command line the script used to start them and doing it manually. There is no bin/start "previous cloud example" I am quite unhappy about that, especially with the rest of the stuff discussed above showing up in the server directory. I was told the latest code will carefully avoid extra directories, but I am not totally sold. Also, the node1/node2 is nearly invisible in the directory and will cause confusion. I think they should be at least prefixed with the example name or something.
          Hide
          Timothy Potter added a comment -

          well if we go with my suggestion about changing the log4j.properties, then we don't need to clone the server directory anymore and can just have:

          example/cloud/node1
          example/cloud/node2
          

          am not totally sold

          What?

          Show
          Timothy Potter added a comment - well if we go with my suggestion about changing the log4j.properties, then we don't need to clone the server directory anymore and can just have: example/cloud/node1 example/cloud/node2 am not totally sold What?
          Hide
          Alexandre Rafalovitch added a comment -

          I am not totally sold that whatever script does to avoid directories inder the server will hold. It feels like a fragile solution to me. But this JIRA will make it less fragile and moving logs out will do so as well.

          Show
          Alexandre Rafalovitch added a comment - I am not totally sold that whatever script does to avoid directories inder the server will hold. It feels like a fragile solution to me. But this JIRA will make it less fragile and moving logs out will do so as well.
          Hide
          Timothy Potter added a comment -

          well then please submit a patch that does something better vs. just calling things fragile.

          Show
          Timothy Potter added a comment - well then please submit a patch that does something better vs. just calling things fragile.
          Hide
          Anshum Gupta added a comment -

          I think we're in a much better shape with the scripts then we were in before this change happened.
          It's iterative and it's evolving. There wasn't any support to automatically restart a previous setup before the scripts happened so I think we're fine as we haven't lost anything and gained a lot of convenience.

          We certainly need updates to the documentation and explain things that need to be manually done until someone automates/scripts.

          Also, I might be missing something here but what makes you think it's fragile?

          Show
          Anshum Gupta added a comment - I think we're in a much better shape with the scripts then we were in before this change happened. It's iterative and it's evolving. There wasn't any support to automatically restart a previous setup before the scripts happened so I think we're fine as we haven't lost anything and gained a lot of convenience. We certainly need updates to the documentation and explain things that need to be manually done until someone automates/scripts. Also, I might be missing something here but what makes you think it's fragile?
          Hide
          Alexandre Rafalovitch added a comment -

          Well, my proposal would be to have server directory locked down and everything else being outside. Which I mentioned before in other JIRAs. I, unfortunately, do not have that level of Solr contributor knowledge yet. Hopefully one day soon, but for now most of my large contributions are outside of Solr codebase.

          On the other hand, I am trying to understand what kind of things will be hard to explain to the new users and - given their tendency to mess up - may cause recovery or consistency issues. A QA-level contribution, you could say. And that intuition comes from spending 3 years supporting multiple versions of multi-million line Java code (BEA Weblogic back in the day) and seeing day after day how even nearly-foolproof configurations do not survive new user's eager attempts at making it work.

          No offense is intended to anybody. I do realize it is a major layout restructuring with the best intent. And it is great overall.

          Show
          Alexandre Rafalovitch added a comment - Well, my proposal would be to have server directory locked down and everything else being outside. Which I mentioned before in other JIRAs. I, unfortunately, do not have that level of Solr contributor knowledge yet. Hopefully one day soon, but for now most of my large contributions are outside of Solr codebase. On the other hand, I am trying to understand what kind of things will be hard to explain to the new users and - given their tendency to mess up - may cause recovery or consistency issues. A QA-level contribution, you could say. And that intuition comes from spending 3 years supporting multiple versions of multi-million line Java code (BEA Weblogic back in the day) and seeing day after day how even nearly-foolproof configurations do not survive new user's eager attempts at making it work. No offense is intended to anybody. I do realize it is a major layout restructuring with the best intent. And it is great overall.
          Hide
          Timothy Potter added a comment -

          Actually, I think I want to go with using:

          solr.log=${solr.solr.home}/../logs/
          

          for the path to the logs directory (in log4j.properties) just in case someone wants to have a core named logs, there won't be any confusion. This results in the following directory structure:

          bin/solr -e techproducts

          example/techproducts/logs/solr.log
          example/techproducts/solr/solr.xml
          example/techproducts/solr/techproducts/data
          example/techproducts/solr/techproducts/conf
          example/techproducts/solr/techproducts/core.properties
          

          bin/solr -e schemaless

          example/schemaless/logs/solr.log
          example/schemaless/solr/solr.xml
          example/schemaless/solr/schemaless/data
          example/schemaless/solr/schemaless/conf
          example/schemaless/solr/schemaless/core.properties
          

          bin/solr -e cloud

          example/cloud/node1/logs/solr.log
          example/cloud/node1/solr/solr.xml
          example/cloud/node1/solr/...
          example/cloud/node2/logs/solr.log
          example/cloud/node2/solr/solr.xml
          example/cloud/node2/solr/...
          

          With that change, if the user just starts Solr without running any examples, then logs will be in server/logs.

          Show
          Timothy Potter added a comment - Actually, I think I want to go with using: solr.log=${solr.solr.home}/../logs/ for the path to the logs directory (in log4j.properties) just in case someone wants to have a core named logs , there won't be any confusion. This results in the following directory structure: bin/solr -e techproducts example/techproducts/logs/solr.log example/techproducts/solr/solr.xml example/techproducts/solr/techproducts/data example/techproducts/solr/techproducts/conf example/techproducts/solr/techproducts/core.properties bin/solr -e schemaless example/schemaless/logs/solr.log example/schemaless/solr/solr.xml example/schemaless/solr/schemaless/data example/schemaless/solr/schemaless/conf example/schemaless/solr/schemaless/core.properties bin/solr -e cloud example/cloud/node1/logs/solr.log example/cloud/node1/solr/solr.xml example/cloud/node1/solr/... example/cloud/node2/logs/solr.log example/cloud/node2/solr/solr.xml example/cloud/node2/solr/... With that change, if the user just starts Solr without running any examples, then logs will be in server/logs.
          Hide
          Timothy Potter added a comment -

          Patch implementing what we discussed for *nix; I'll wait until we're in agreement on the experience before porting to solr.cmd for Windows.

          Show
          Timothy Potter added a comment - Patch implementing what we discussed for *nix; I'll wait until we're in agreement on the experience before porting to solr.cmd for Windows.
          Hide
          Timothy Potter added a comment -

          ugh! looking like we may need to use a different resources/log4j.properties for examples since setting:

          solr.log=${solr.solr.home}/../logs/
          

          results in the following error if you start Solr the old-fashioned way:

          $ java -jar start.jar
          log4j:ERROR setFile(null,true) call failed.
          java.io.FileNotFoundException: /../logs/solr.log (No such file or directory)
          	at java.io.FileOutputStream.open(Native Method)
          	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
          	at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
          	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
          	at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
          	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
          

          so that's not good ...

          Show
          Timothy Potter added a comment - ugh! looking like we may need to use a different resources/log4j.properties for examples since setting: solr.log=${solr.solr.home}/../logs/ results in the following error if you start Solr the old-fashioned way: $ java -jar start.jar log4j:ERROR setFile( null , true ) call failed. java.io.FileNotFoundException: /../logs/solr.log (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:133) at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) so that's not good ...
          Hide
          Hoss Man added a comment -

          ugh! looking like we may need to use a different resources/log4j.properties for examples ...

          might be for the best anyway: maybe leave the "production" log4j.properties lean with only WARN/ERROR enabled for most packages, while the example logging file could be set to INFO?

          (I'm not suggesting we make them substantively different right now as part of this issue - just pointing out there may be value in having them be distinct files)

          Show
          Hoss Man added a comment - ugh! looking like we may need to use a different resources/log4j.properties for examples ... might be for the best anyway: maybe leave the "production" log4j.properties lean with only WARN/ERROR enabled for most packages, while the example logging file could be set to INFO? (I'm not suggesting we make them substantively different right now as part of this issue - just pointing out there may be value in having them be distinct files)
          Hide
          Timothy Potter added a comment -

          Thanks Hoss Man - I'll go with that solution (example/resources/log4j.properties). Should have something to commit shortly.

          Show
          Timothy Potter added a comment - Thanks Hoss Man - I'll go with that solution (example/resources/log4j.properties). Should have something to commit shortly.
          Hide
          ASF subversion and git services added a comment -

          Commit 1644799 from Timothy Potter in branch 'dev/trunk'
          [ https://svn.apache.org/r1644799 ]

          SOLR-6833: Examples started with bin/solr -e should use a solr.solr.home directory under the example directory instead of server/solr.

          Show
          ASF subversion and git services added a comment - Commit 1644799 from Timothy Potter in branch 'dev/trunk' [ https://svn.apache.org/r1644799 ] SOLR-6833 : Examples started with bin/solr -e should use a solr.solr.home directory under the example directory instead of server/solr.
          Hide
          ASF subversion and git services added a comment -

          Commit 1644801 from Timothy Potter in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1644801 ]

          SOLR-6833: Examples started with bin/solr -e should use a solr.solr.home directory under the example directory instead of server/solr.

          Show
          ASF subversion and git services added a comment - Commit 1644801 from Timothy Potter in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1644801 ] SOLR-6833 : Examples started with bin/solr -e should use a solr.solr.home directory under the example directory instead of server/solr.
          Hide
          Timothy Potter added a comment -

          I think this one is good on trunk and 5x now, but would be good if some others could kick the tires a bit in case I overlooked something.

          Show
          Timothy Potter added a comment - I think this one is good on trunk and 5x now, but would be good if some others could kick the tires a bit in case I overlooked something.
          Hide
          Alexandre Rafalovitch added a comment -

          I found a cool command I did not see while reading the documentation before solr -i/-info. It's mentioned in passing in solr stop -help but could be nice to be mentioned in top-level help as well. It's very useful to check what's running.

          Show
          Alexandre Rafalovitch added a comment - I found a cool command I did not see while reading the documentation before solr -i/-info . It's mentioned in passing in solr stop -help but could be nice to be mentioned in top-level help as well. It's very useful to check what's running.
          Hide
          Alexandre Rafalovitch added a comment -

          Looks good. I hit a script bombing out on mis-spelt name, so opened a SOLR-6848 for that.

          The rest here are just random thoughts from using the scripts, feel free to ignore if this has already been discussed.

          1. The example run will generate console.log, even though log4j for example does not send anything to it. So the file is always empty.
          2. There is also solr_gc.log for the examples, I am not sure if that's a big deal. It grows kind of fast, but so does the main log.
          3. In the script, if the port is busy, is says:

            Port 8983 is already being used by another process (pid: 13221)

            Could be even more - first user - nicer to add after: Use -p flag to start with a different port.

          Show
          Alexandre Rafalovitch added a comment - Looks good. I hit a script bombing out on mis-spelt name, so opened a SOLR-6848 for that. The rest here are just random thoughts from using the scripts, feel free to ignore if this has already been discussed. The example run will generate console.log, even though log4j for example does not send anything to it. So the file is always empty. There is also solr_gc.log for the examples, I am not sure if that's a big deal. It grows kind of fast, but so does the main log. In the script, if the port is busy, is says: Port 8983 is already being used by another process (pid: 13221) Could be even more - first user - nicer to add after: Use -p flag to start with a different port .
          Hide
          Anshum Gupta added a comment -

          I think this change started bundling 'gettingstarted' collection out of the box.

          Show
          Anshum Gupta added a comment - I think this change started bundling 'gettingstarted' collection out of the box.
          Hide
          Anshum Gupta added a comment -

          It's just the build and my fault. I ran the example from here once... and then did an "ant clean package"... that seems to have bundled /cloud with the gettingstarted data with it.

          Show
          Anshum Gupta added a comment - It's just the build and my fault. I ran the example from here once... and then did an "ant clean package"... that seems to have bundled /cloud with the gettingstarted data with it.
          Hide
          ASF subversion and git services added a comment -

          Commit 1645737 from Timothy Potter in branch 'dev/trunk'
          [ https://svn.apache.org/r1645737 ]

          SOLR-6833: clean should remove example directories created by running bin/solr -e foo

          Show
          ASF subversion and git services added a comment - Commit 1645737 from Timothy Potter in branch 'dev/trunk' [ https://svn.apache.org/r1645737 ] SOLR-6833 : clean should remove example directories created by running bin/solr -e foo
          Hide
          ASF subversion and git services added a comment -

          Commit 1645741 from Timothy Potter in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1645741 ]

          SOLR-6833: clean should remove example directories created by running bin/solr -e foo

          Show
          ASF subversion and git services added a comment - Commit 1645741 from Timothy Potter in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1645741 ] SOLR-6833 : clean should remove example directories created by running bin/solr -e foo
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.

            People

            • Assignee:
              Timothy Potter
              Reporter:
              Hoss Man
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development