Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10735

Solr is broken when directory with spaces used on Windows

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.5
    • Fix Version/s: 6.6, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Uwe Schindler mentioned this in the 6.6 RC1 voting thread:

      The startup script (Windows at least) again does not work with whitepsace directory names, which is standard on Windows. It does give an error message not while server startup, but when trying to create the techproducts core. I am about to open issue.
      
      1. Screenshot from 2017-05-24 21-00-29.png
        41 kB
        Ishan Chattopadhyaya
      2. Screenshot from 2017-05-27 01-49-43.png
        68 kB
        Ishan Chattopadhyaya
      3. Screenshot from 2017-05-27 02-16-04.png
        70 kB
        Ishan Chattopadhyaya
      4. screenshot-with-patch.png
        154 kB
        Uwe Schindler
      5. SOLR-10735.patch
        0.8 kB
        Ishan Chattopadhyaya

        Activity

        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        While I have some time before I spin the next RC, I'm looking to reproduce this. I may or may not be able to fix it in time for the next RC, unless someone feels this should be a blocker.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - While I have some time before I spin the next RC, I'm looking to reproduce this. I may or may not be able to fix it in time for the next RC, unless someone feels this should be a blocker.
        Hide
        janhoy Jan Høydahl added a comment -

        +1 to blocker. A regression in such a central feature as bin\solr.cmd create should probably be fixed before a release!

        Show
        janhoy Jan Høydahl added a comment - +1 to blocker. A regression in such a central feature as bin\solr.cmd create should probably be fixed before a release!
        Hide
        janhoy Jan Høydahl added a comment -

        I tried to reproduce as well but did not get to the bottom of it.

        Show
        janhoy Jan Høydahl added a comment - I tried to reproduce as well but did not get to the bottom of it.
        Hide
        thetaphi Uwe Schindler added a comment -

        Hi,
        Jan Høydahl does it reproduce for you? (I don't understand what you mean with "but did not get to the bottom of it".
        Interestingly only creating the techproducts example does not work in the RC, but starting solr works.
        In the Solr 6-5 as shipped currently, the Solr server does not even start (so something changed inbetween).
        Uwe

        Show
        thetaphi Uwe Schindler added a comment - Hi, Jan Høydahl does it reproduce for you? (I don't understand what you mean with "but did not get to the bottom of it". Interestingly only creating the techproducts example does not work in the RC, but starting solr works. In the Solr 6-5 as shipped currently, the Solr server does not even start (so something changed inbetween). Uwe
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Indeed, I tried a lot and I couldn't reproduce it today. Uwe's comment makes it clear that regular operations like start/stop, create_collection etc. are not affected; only techproducts example is broken (which I didn't test). I've attached a screenshot of my attempt to reproduce.

        Based on this, I think this doesn't need to be a blocker. If someone disagrees, please mark it as a blocker.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Indeed, I tried a lot and I couldn't reproduce it today. Uwe's comment makes it clear that regular operations like start/stop, create_collection etc. are not affected; only techproducts example is broken (which I didn't test). I've attached a screenshot of my attempt to reproduce. Based on this, I think this doesn't need to be a blocker. If someone disagrees, please mark it as a blocker.
        Hide
        janhoy Jan Høydahl added a comment -

        My first test was a self-built master version but I got into write permission trouble...

        I just tested on Win Server 2012 with a Solr 6.5.1, and successfully did solr start, solr create, and even solr -e techproducts, when running from a folder with space in it. I also tried solr create -c foo -d folder and solr start -s "C:\path with space" and they both work.

        Then I tried solr create -c foo -d "C:\path with spaces\to\conf", and that one failed with error

        ERROR: Specified config directory C:\Users\jho\Desktop\solr not found in C:\Users\jho\Desktop\solr with space\server\sol
        r\configsets
        

        I don't know if that particular command used to work pre 6.5?

        The script code in question is

          "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% -Dsolr.install.dir="%SOLR_TIP%" ^
            -Dlog4j.configuration="file:%DEFAULT_SERVER_DIR%\scripts\cloud-scripts\log4j.properties" ^
            -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
            org.apache.solr.util.SolrCLI create_core -name !CREATE_NAME! -solrUrl !SOLR_URL_SCHEME!://%SOLR_TOOL_HOST%:!CREATE_PORT!/solr ^
            -confdir !CREATE_CONFDIR! -configsetsDir "%SOLR_TIP%\server\solr\configsets"
        

        I think the -confdir !CREATE_CONFDIR! is the sinner, it should be quoted, not?

        Show
        janhoy Jan Høydahl added a comment - My first test was a self-built master version but I got into write permission trouble... I just tested on Win Server 2012 with a Solr 6.5.1, and successfully did solr start , solr create , and even solr -e techproducts , when running from a folder with space in it. I also tried solr create -c foo -d folder and solr start -s "C:\path with space" and they both work. Then I tried solr create -c foo -d "C:\path with spaces\to\conf" , and that one failed with error ERROR: Specified config directory C:\Users\jho\Desktop\solr not found in C:\Users\jho\Desktop\solr with space\server\sol r\configsets I don't know if that particular command used to work pre 6.5? The script code in question is "%JAVA%" %SOLR_SSL_OPTS% %AUTHC_OPTS% %SOLR_ZK_CREDS_AND_ACLS% -Dsolr.install.dir= "%SOLR_TIP%" ^ -Dlog4j.configuration= "file:%DEFAULT_SERVER_DIR%\scripts\cloud-scripts\log4j.properties" ^ -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^ org.apache.solr.util.SolrCLI create_core -name !CREATE_NAME! -solrUrl !SOLR_URL_SCHEME!: //%SOLR_TOOL_HOST%:!CREATE_PORT!/solr ^ -confdir !CREATE_CONFDIR! -configsetsDir "%SOLR_TIP%\server\solr\configsets" I think the -confdir !CREATE_CONFDIR! is the sinner, it should be quoted, not?
        Hide
        janhoy Jan Høydahl added a comment -

        Uwe Schindler Since I was not able to reproduce for techproducts example, can you give the exact steps for reproduction, including error messages?

        Show
        janhoy Jan Høydahl added a comment - Uwe Schindler Since I was not able to reproduce for techproducts example, can you give the exact steps for reproduction, including error messages?
        Hide
        thetaphi Uwe Schindler added a comment -

        Here is my output with the Solr 6.6.0 release candidate:

        C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin>solr.cmd start -e techproducts
        Creating Solr home directory C:\Users\Uwe Schindler\Desktop\solr-6.6.0\example\techproducts\solr
        
        Starting up Solr on port 8983 using command:
        C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin\solr.cmd start -p 8983 -s "C:\Users\Uwe Schindler\Desktop\solr-6.6.0\example\techproducts\solr"
        
        
        ERROR: Failed to start Solr using command: C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin\solr.cmd start -p 8983 -s "C:\Users\Uwe Schindler\Desktop\solr-6.6.0\example\techproducts\solr" Exception : org.apache.commons.exec.ExecuteException: Execution failed (Exit value: -559038737. Caused by java.io.IOException: Cannot run program "C:\Users\Uwe" (in directory "."): CreateProcess error=193, %1 ist keine zulässige Win32-Anwendung)
        

        This is when starting without example (in Solr 6.5.1 this failed, too):

        C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin>solr.cmd start
        Waiting up to 30 to see Solr running on port 8983
        Started Solr server on port 8983. Happy searching!
        

        So the general startup script works, but booting the techexample breaks. I think it is the (IMHO broken commons-commandline stuff, I hate it - damns broken, sorry!). Java 8 has a phantastic ProcessBuilder API, why do we need such sh*t like command line libs for stuff that works OOB??

        Show
        thetaphi Uwe Schindler added a comment - Here is my output with the Solr 6.6.0 release candidate: C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin>solr.cmd start -e techproducts Creating Solr home directory C:\Users\Uwe Schindler\Desktop\solr-6.6.0\example\techproducts\solr Starting up Solr on port 8983 using command: C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin\solr.cmd start -p 8983 -s "C:\Users\Uwe Schindler\Desktop\solr-6.6.0\example\techproducts\solr" ERROR: Failed to start Solr using command: C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin\solr.cmd start -p 8983 -s "C:\Users\Uwe Schindler\Desktop\solr-6.6.0\example\techproducts\solr" Exception : org.apache.commons.exec.ExecuteException: Execution failed (Exit value: -559038737. Caused by java.io.IOException: Cannot run program "C:\Users\Uwe" (in directory "."): CreateProcess error=193, %1 ist keine zulässige Win32-Anwendung) This is when starting without example (in Solr 6.5.1 this failed, too): C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin>solr.cmd start Waiting up to 30 to see Solr running on port 8983 Started Solr server on port 8983. Happy searching! So the general startup script works, but booting the techexample breaks. I think it is the (IMHO broken commons-commandline stuff, I hate it - damns broken, sorry!). Java 8 has a phantastic ProcessBuilder API, why do we need such sh*t like command line libs for stuff that works OOB??
        Hide
        tomasflobbe Tomás Fernández Löbbe added a comment -

        I was also able to start Solr and techproducts on a directory with whitespaces. Don't know much about the start scripts, but can the problem be that Uwe's username has whitespaces?

        Show
        tomasflobbe Tomás Fernández Löbbe added a comment - I was also able to start Solr and techproducts on a directory with whitespaces. Don't know much about the start scripts, but can the problem be that Uwe's username has whitespaces?
        Hide
        thetaphi Uwe Schindler added a comment - - edited

        ...or Windows 10? I can reproduce this with 6.6 and earlier versions (6.3) of Solr work - except 6.5, which already fails when staring the server (without techproducts).

        The error message suggests that the buggy commons-process library for process executions somehow tokenizes at whitespace.

        Show
        thetaphi Uwe Schindler added a comment - - edited ...or Windows 10? I can reproduce this with 6.6 and earlier versions (6.3) of Solr work - except 6.5, which already fails when staring the server (without techproducts). The error message suggests that the buggy commons-process library for process executions somehow tokenizes at whitespace.
        Hide
        tomasflobbe Tomás Fernández Löbbe added a comment -

        I tested with Windows 10 too.

        Show
        tomasflobbe Tomás Fernández Löbbe added a comment - I tested with Windows 10 too.
        Hide
        thetaphi Uwe Schindler added a comment -
        Failed to start Solr using command: C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin\solr.cmd start -p 8983 -s "C:\Users\Uwe Schindler\Desktop\solr-6.6.0\example\techproducts\solr"
        

        It tries to run the solr.cmd file without adding " around the command itsself. There is nothing more to say.

        Show
        thetaphi Uwe Schindler added a comment - Failed to start Solr using command: C:\Users\Uwe Schindler\Desktop\solr-6.6.0\bin\solr.cmd start -p 8983 -s "C:\Users\Uwe Schindler\Desktop\solr-6.6.0\example\techproducts\solr" It tries to run the solr.cmd file without adding " around the command itsself. There is nothing more to say.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        It worked fine for me with for the path similar to Uwe's. I'm on Windows 10.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - It worked fine for me with for the path similar to Uwe's. I'm on Windows 10.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Same as above when I try this using the RC3 zip file.

        Uwe, what do you think we should do? Is there a quick fix that you propose that we can make (to the solr.cmd) to make it work for users who might be facing issues like you are facing?

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Same as above when I try this using the RC3 zip file. Uwe, what do you think we should do? Is there a quick fix that you propose that we can make (to the solr.cmd) to make it work for users who might be facing issues like you are facing?
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        My E: is an NTFS drive. Could it be that some other filesystem (FAT32) could be causing problems?

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - My E: is an NTFS drive. Could it be that some other filesystem (FAT32) could be causing problems?
        Hide
        janhoy Jan Høydahl added a comment -

        I think this is the offender: https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/util/SolrCLI.java#L2944
        On normal solr start -script is not passed. but for example it is, and as u can see, not quoted.
        Instead of ugly string magic we should use CommandLine.addArgument() or JRE builtin launcher..

        Show
        janhoy Jan Høydahl added a comment - I think this is the offender: https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/util/SolrCLI.java#L2944 On normal solr start -script is not passed. but for example it is, and as u can see, not quoted. Instead of ugly string magic we should use CommandLine.addArgument() or JRE builtin launcher..
        Hide
        tomasflobbe Tomás Fernández Löbbe added a comment -

        Same here, tested with Windows 10 Home on "C:\Users\Tomas Fernandez Lobb\Downloads\solr-6.6.0" and it worked for me. Not sure what's going on.

        Instead of ugly string magic we should use CommandLine.addArgument() or JRE builtin launcher..

        +1, but how do we test? Only Uwe was able to reproduce so far, any ideas why?

        Show
        tomasflobbe Tomás Fernández Löbbe added a comment - Same here, tested with Windows 10 Home on "C:\Users\Tomas Fernandez Lobb\Downloads\solr-6.6.0" and it worked for me. Not sure what's going on. Instead of ugly string magic we should use CommandLine.addArgument() or JRE builtin launcher.. +1, but how do we test? Only Uwe was able to reproduce so far, any ideas why?
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        I've tried putting quotes around the Solr command in the SolrCLI place which Jan pointed to, and it works for me. So, if that works for Uwe as well, we're all good. The patch for branch_6_6 is attached.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - I've tried putting quotes around the Solr command in the SolrCLI place which Jan pointed to, and it works for me. So, if that works for Uwe as well, we're all good. The patch for branch_6_6 is attached.
        Hide
        janhoy Jan Høydahl added a comment -

        I tried on my Win10 Pro Microsoft Windows [Version 10.0.10240], Java 1.8.0_131, running in cmd.exe, and could not reproduce:

        C:\Users\janms\apache solr\solr-6.5.1>bin\solr -e techproducts -V
        [...]
        Starting up Solr on port 8983 using command:
        C:\Users\janms\apache solr\solr-6.5.1\bin\solr.cmd start -p 8983 -s "C:\Users\janms\apache solr\solr-6.5.1\example\techproducts\solr"
        

        The printout above suggests that this is the string parsed by common exec, but it succeeds all the same. But if I try to copy-paste that same string into CMD.exe, I get

        C:\Users\janms\apache solr\solr-6.5.1>C:\Users\janms\apache solr\solr-6.5.1\bin\solr.cmd start -p 8983 -s "C:\Users\janms\apache solr\solr-6.5.1\example\techproducts\solr"
        'C:\Users\janms\apache' is not recognized as an internal or external command, operable program or batch file.
        
        Show
        janhoy Jan Høydahl added a comment - I tried on my Win10 Pro Microsoft Windows [Version 10.0.10240], Java 1.8.0_131, running in cmd.exe, and could not reproduce: C:\Users\janms\apache solr\solr-6.5.1>bin\solr -e techproducts -V [...] Starting up Solr on port 8983 using command: C:\Users\janms\apache solr\solr-6.5.1\bin\solr.cmd start -p 8983 -s "C:\Users\janms\apache solr\solr-6.5.1\example\techproducts\solr" The printout above suggests that this is the string parsed by common exec, but it succeeds all the same. But if I try to copy-paste that same string into CMD.exe, I get C:\Users\janms\apache solr\solr-6.5.1>C:\Users\janms\apache solr\solr-6.5.1\bin\solr.cmd start -p 8983 -s "C:\Users\janms\apache solr\solr-6.5.1\example\techproducts\solr" 'C:\Users\janms\apache' is not recognized as an internal or external command, operable program or batch file.
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        I tried the PowerShell, but couldn't reproduce as well.

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - I tried the PowerShell, but couldn't reproduce as well.
        Hide
        thetaphi Uwe Schindler added a comment - - edited

        Maybe it has also to do with the german language, but IMHO the attached patch is the correct way to fix this.

        I'd suggest that we change smoketester to use a folder with whitespace. Should I open an issue for that? This way we can catch bugs like this faster – although I never found a way how to run smoketester on Windows without cygwin, so not useable on Policeman Jenkins.

        Show
        thetaphi Uwe Schindler added a comment - - edited Maybe it has also to do with the german language, but IMHO the attached patch is the correct way to fix this. I'd suggest that we change smoketester to use a folder with whitespace. Should I open an issue for that? This way we can catch bugs like this faster – although I never found a way how to run smoketester on Windows without cygwin, so not useable on Policeman Jenkins.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 45b26e322f1e173c8a19f07700e64daa5475da84 in lucene-solr's branch refs/heads/master from Ishan Chattopadhyaya
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=45b26e3 ]

        SOLR-10735: Fixing windows space directory issue

        Show
        jira-bot ASF subversion and git services added a comment - Commit 45b26e322f1e173c8a19f07700e64daa5475da84 in lucene-solr's branch refs/heads/master from Ishan Chattopadhyaya [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=45b26e3 ] SOLR-10735 : Fixing windows space directory issue
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 995291df788ed75d8430f40bb99cc668de86c6aa in lucene-solr's branch refs/heads/branch_6x from Ishan Chattopadhyaya
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=995291d ]

        SOLR-10735: Fixing windows space directory issue

        Show
        jira-bot ASF subversion and git services added a comment - Commit 995291df788ed75d8430f40bb99cc668de86c6aa in lucene-solr's branch refs/heads/branch_6x from Ishan Chattopadhyaya [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=995291d ] SOLR-10735 : Fixing windows space directory issue
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 732e8331cf984c0ebe02e20f70daacd82817bd4e in lucene-solr's branch refs/heads/branch_6_6 from Ishan Chattopadhyaya
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=732e833 ]

        SOLR-10735: Fixing windows space directory issue

        Show
        jira-bot ASF subversion and git services added a comment - Commit 732e8331cf984c0ebe02e20f70daacd82817bd4e in lucene-solr's branch refs/heads/branch_6_6 from Ishan Chattopadhyaya [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=732e833 ] SOLR-10735 : Fixing windows space directory issue
        Hide
        thetaphi Uwe Schindler added a comment -

        I rebuilt a release with the attached patch. For me this starts in the same setup as before:

        Show
        thetaphi Uwe Schindler added a comment - I rebuilt a release with the attached patch. For me this starts in the same setup as before:
        Hide
        ichattopadhyaya Ishan Chattopadhyaya added a comment -

        Thanks Uwe Schindler! Seems like it was the German locale, indeed!

        Show
        ichattopadhyaya Ishan Chattopadhyaya added a comment - Thanks Uwe Schindler ! Seems like it was the German locale, indeed!

          People

          • Assignee:
            ichattopadhyaya Ishan Chattopadhyaya
            Reporter:
            ichattopadhyaya Ishan Chattopadhyaya
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development