Solr
  1. Solr
  2. SOLR-7921

techproducts example broken; dir with spaces; Linux/Windows

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.3
    • Fix Version/s: 5.3, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      After Uwe Schindler reported that the error with spaces in dir names (SOLR-7016) has reappeared in 5.3.0 RC1, I checked the same on Linux and found it is broken. Here's the console log:

      [ishan@localhost solr 530]$ bin/solr start -e techproducts
      Creating Solr home directory /home/ishan/solr 530/example/techproducts/solr
      
      Starting up Solr on port 8983 using command:
      bin/solr start -p 8983 -s "example/techproducts/solr"
      
      NOTE: Please install lsof as this script needs it to determine if Solr is listening on port 8983.
      
      Started Solr server on port 8983 (pid=2404). Happy searching!
      
      
      Setup new core instance directory:
      /home/ishan/solr 530/example/techproducts/solr/techproducts
      
      Creating new core 'techproducts' using command:
      http://localhost:8983/solr/admin/cores?action=CREATE&name=techproducts&instanceDir=techproducts
      
      {
        "responseHeader":{
          "status":0,
          "QTime":2671},
        "core":"techproducts"}
      
      
      Indexing tech product example docs from /home/ishan/solr 530/example/exampledocs
      Error: Unable to access jarfile /home/ishan/solr
      
      ERROR: Process exited with an error: 1 (Exit value: 1)
      
      1. SOLR-7921.patch
        4 kB
        Ishan Chattopadhyaya
      2. SOLR-7921.patch
        2 kB
        Ishan Chattopadhyaya
      3. SOLR-7921.patch
        2 kB
        Ishan Chattopadhyaya
      4. SOLR-7921.patch
        0.9 kB
        Ishan Chattopadhyaya
      5. SOLR-7921.patch
        0.8 kB
        Ishan Chattopadhyaya

        Issue Links

          Activity

          Hide
          Erik Hatcher added a comment -

          but bin/post works

          # -e techproducts fail...
          $ bin/solr start -p 8888 -e techproducts
             ...
          
          Indexing tech product example docs from /Users/erikhatcher/Downloads/solr 5.3.0-RC1/example/exampledocs
          Error: Unable to access jarfile /Users/erikhatcher/Downloads/solr
          
          ERROR: Process exited with an error: 1 (Exit value: 1)
          
          # bin/post works...
          
          $ bin/post -p 8888 -c techproducts example/exampledocs/*.xml
          java -classpath /Users/erikhatcher/Downloads/solr 5.3.0-RC1/dist/solr-core-5.3.0.jar -Dauto=yes -Dport=8888 -Dc=techproducts -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/gb18030-example.xml example/exampledocs/hd.xml example/exampledocs/ipod_other.xml example/exampledocs/ipod_video.xml example/exampledocs/manufacturers.xml example/exampledocs/mem.xml example/exampledocs/money.xml example/exampledocs/monitor.xml example/exampledocs/monitor2.xml example/exampledocs/mp500.xml example/exampledocs/sd500.xml example/exampledocs/solr.xml example/exampledocs/utf8-example.xml example/exampledocs/vidcard.xml
          SimplePostTool version 5.0.0
          
             ...
          
          14 files indexed.
          COMMITting Solr index changes to http://localhost:8888/solr/techproducts/update...
          Time spent: 0:00:00.248
          

          Now we just need SOLR-6994

          Show
          Erik Hatcher added a comment - but bin/post works # -e techproducts fail... $ bin/solr start -p 8888 -e techproducts ... Indexing tech product example docs from /Users/erikhatcher/Downloads/solr 5.3.0-RC1/example/exampledocs Error: Unable to access jarfile /Users/erikhatcher/Downloads/solr ERROR: Process exited with an error: 1 (Exit value: 1) # bin/post works... $ bin/post -p 8888 -c techproducts example/exampledocs/*.xml java -classpath /Users/erikhatcher/Downloads/solr 5.3.0-RC1/dist/solr-core-5.3.0.jar -Dauto=yes -Dport=8888 -Dc=techproducts -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/gb18030-example.xml example/exampledocs/hd.xml example/exampledocs/ipod_other.xml example/exampledocs/ipod_video.xml example/exampledocs/manufacturers.xml example/exampledocs/mem.xml example/exampledocs/money.xml example/exampledocs/monitor.xml example/exampledocs/monitor2.xml example/exampledocs/mp500.xml example/exampledocs/sd500.xml example/exampledocs/solr.xml example/exampledocs/utf8-example.xml example/exampledocs/vidcard.xml SimplePostTool version 5.0.0 ... 14 files indexed. COMMITting Solr index changes to http: //localhost:8888/solr/techproducts/update... Time spent: 0:00:00.248 Now we just need SOLR-6994
          Hide
          Ishan Chattopadhyaya added a comment -

          In my previously attached patch, the java -jar command uses double quotes around the jar file. This seems to work fine in shell (bash and sh), but still fails through the executor.execute(). Attaching a new patch that bypasses that code path altogether.

          Seems to work, based on a manual test. Will write a unit test for this next, unless we can actually want to do the SOLR-6994 before the release.

          Show
          Ishan Chattopadhyaya added a comment - In my previously attached patch, the java -jar command uses double quotes around the jar file. This seems to work fine in shell (bash and sh), but still fails through the executor.execute(). Attaching a new patch that bypasses that code path altogether. Seems to work, based on a manual test. Will write a unit test for this next, unless we can actually want to do the SOLR-6994 before the release.
          Hide
          Timothy Potter added a comment -

          Let's just get rid of the following block and invoke the SimplePostTool directly for now, i.e. remove this if and just go with the else part:

          if (postJarFile.isFile()) {
                      String javaHome = System.getProperty("java.home");
                      String java = javaHome+"/bin/java";
                      String postCmd = String.format(Locale.ROOT, "%s -Durl=\"%s\" -jar %s \"%s\"/*.xml",
                          java, updateUrl, postJarFile.getAbsolutePath(), exampledocsDir.getAbsolutePath());
                      executor.execute(org.apache.commons.exec.CommandLine.parse(postCmd));
                    } else {
                    ....
          
          Show
          Timothy Potter added a comment - Let's just get rid of the following block and invoke the SimplePostTool directly for now, i.e. remove this if and just go with the else part: if (postJarFile.isFile()) { String javaHome = System .getProperty( "java.home" ); String java = javaHome+ "/bin/java" ; String postCmd = String .format(Locale.ROOT, "%s -Durl=\" %s\ " -jar %s \" %s\ "/*.xml" , java, updateUrl, postJarFile.getAbsolutePath(), exampledocsDir.getAbsolutePath()); executor.execute(org.apache.commons.exec.CommandLine.parse(postCmd)); } else { ....
          Hide
          Erik Hatcher added a comment - - edited

          Even with SOLR-6994, SolrCLI would need modification to call out to the script rather than use post.jar directly.

          Can't we just include ant.jar and launch everything that way?

          Show
          Erik Hatcher added a comment - - edited Even with SOLR-6994 , SolrCLI would need modification to call out to the script rather than use post.jar directly. Can't we just include ant.jar and launch everything that way?
          Hide
          Ishan Chattopadhyaya added a comment -

          +1 Timothy Potter. Updating the patch, removing the if block.

          Writing a unit test for dir with spaces is proving a bit tricky, due to some test-framework issues.

          Show
          Ishan Chattopadhyaya added a comment - +1 Timothy Potter . Updating the patch, removing the if block. Writing a unit test for dir with spaces is proving a bit tricky, due to some test-framework issues.
          Hide
          Ishan Chattopadhyaya added a comment -

          Removing the @Ignore for the corresponding existing test.

          Show
          Ishan Chattopadhyaya added a comment - Removing the @Ignore for the corresponding existing test.
          Hide
          Ishan Chattopadhyaya added a comment -

          The post.jar is no longer needed in the test class. Removing it from the test.

          Show
          Ishan Chattopadhyaya added a comment - The post.jar is no longer needed in the test class. Removing it from the test.
          Hide
          Timothy Potter added a comment -

          looks good Ishan, I'll commit -thanks

          Show
          Timothy Potter added a comment - looks good Ishan, I'll commit -thanks
          Hide
          Timothy Potter added a comment -

          Uwe Schindler can you apply this patch and run the unit test and try: bin\solr.cmd -e techproducts in your Windows environment before I commit please?

          Show
          Timothy Potter added a comment - Uwe Schindler can you apply this patch and run the unit test and try: bin\solr.cmd -e techproducts in your Windows environment before I commit please?
          Hide
          Erik Hatcher added a comment -

          Nicely done Ishan Chattopadhyaya.

          "Simple*" things should be simple(r).

          Show
          Erik Hatcher added a comment - Nicely done Ishan Chattopadhyaya . "Simple*" things should be simple(r).
          Hide
          Uwe Schindler added a comment -

          Works on Windows (trunk branch):

          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr start -e techproducts
          Creating Solr home directory C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr
          
          Starting up Solr on port 8983 using command:
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin\solr.cmd start -p 8983 -s "C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr"
          
          Waiting up to 30 to see Solr running on port 8983
          Started Solr server on port 8983. Happy searching!
          
          Setup new core instance directory:
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr\techproducts
          
          Creating new core 'techproducts' using command:
          http://localhost:8983/solr/admin/cores?action=CREATE&name=techproducts&instanceDir=techproducts
          
          {
            "responseHeader":{
              "status":0,
              "QTime":1126},
            "core":"techproducts"}
          
          
          Indexing tech product example docs from C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\exampledocs
          SimplePostTool version 5.0.0
          Posting files to [base] url http://localhost:8983/solr/techproducts/update using content-type application/xml...
          POSTing file gb18030-example.xml to [base]
          POSTing file hd.xml to [base]
          POSTing file ipod_other.xml to [base]
          POSTing file ipod_video.xml to [base]
          POSTing file manufacturers.xml to [base]
          POSTing file mem.xml to [base]
          POSTing file money.xml to [base]
          POSTing file monitor.xml to [base]
          POSTing file monitor2.xml to [base]
          POSTing file mp500.xml to [base]
          POSTing file sd500.xml to [base]
          POSTing file solr.xml to [base]
          POSTing file utf8-example.xml to [base]
          POSTing file vidcard.xml to [base]
          14 files indexed.
          COMMITting Solr index changes to http://localhost:8983/solr/techproducts/update...
          Time spent: 0:00:00.293
          
          Solr techproducts example launched successfully. Direct your Web browser to http://localhost:8983/solr to visit the Solr Admin UI
          
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>
          
          Show
          Uwe Schindler added a comment - Works on Windows (trunk branch): C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr start -e techproducts Creating Solr home directory C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr Starting up Solr on port 8983 using command: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin\solr.cmd start -p 8983 -s "C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr" Waiting up to 30 to see Solr running on port 8983 Started Solr server on port 8983. Happy searching! Setup new core instance directory: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr\techproducts Creating new core 'techproducts' using command: http://localhost:8983/solr/admin/cores?action=CREATE&name=techproducts&instanceDir=techproducts { "responseHeader":{ "status":0, "QTime":1126}, "core":"techproducts"} Indexing tech product example docs from C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\exampledocs SimplePostTool version 5.0.0 Posting files to [base] url http://localhost:8983/solr/techproducts/update using content-type application/xml... POSTing file gb18030-example.xml to [base] POSTing file hd.xml to [base] POSTing file ipod_other.xml to [base] POSTing file ipod_video.xml to [base] POSTing file manufacturers.xml to [base] POSTing file mem.xml to [base] POSTing file money.xml to [base] POSTing file monitor.xml to [base] POSTing file monitor2.xml to [base] POSTing file mp500.xml to [base] POSTing file sd500.xml to [base] POSTing file solr.xml to [base] POSTing file utf8-example.xml to [base] POSTing file vidcard.xml to [base] 14 files indexed. COMMITting Solr index changes to http://localhost:8983/solr/techproducts/update... Time spent: 0:00:00.293 Solr techproducts example launched successfully. Direct your Web browser to http://localhost:8983/solr to visit the Solr Admin UI C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>
          Hide
          Uwe Schindler added a comment -

          It's also repeatable, so stopping and restarting with same command also works.

          The issue I had was because I killed the server using windows task manager. Because of this it tried to recreate the core but failed. It seems that the core is persistend on disk only on "clean" shutdown. When I tried to reexceute the command, it tried to recreate the "half-existing" core and failed, because the IndexReader/Writer was already open. We should improve that (if you kill -9 solr, it should at least keep the cores already created).

          Show
          Uwe Schindler added a comment - It's also repeatable, so stopping and restarting with same command also works. The issue I had was because I killed the server using windows task manager. Because of this it tried to recreate the core but failed. It seems that the core is persistend on disk only on "clean" shutdown. When I tried to reexceute the command, it tried to recreate the "half-existing" core and failed, because the IndexReader/Writer was already open. We should improve that (if you kill -9 solr, it should at least keep the cores already created).
          Hide
          ASF subversion and git services added a comment -

          Commit 1695728 from Timothy Potter in branch 'dev/trunk'
          [ https://svn.apache.org/r1695728 ]

          SOLR-7921: The techproducts example fails when running in a directory that contains spaces.

          Show
          ASF subversion and git services added a comment - Commit 1695728 from Timothy Potter in branch 'dev/trunk' [ https://svn.apache.org/r1695728 ] SOLR-7921 : The techproducts example fails when running in a directory that contains spaces.
          Hide
          Uwe Schindler added a comment -

          Even with SOLR-6994, SolrCLI would need modification to call out to the script rather than use post.jar directly. Can't we just include ant.jar and launch everything that way?

          I don't think this is a good idea. The problem is that you are building the command line by adding the args separated by spaces. Since Java 5 and even better with Java 8, you can use ProcessBuilder where you can add args and executable in a clean way. No need for the old-style ANT stuff (that is mainly there to allow running ANT on pre-Java-5).

          Show
          Uwe Schindler added a comment - Even with SOLR-6994 , SolrCLI would need modification to call out to the script rather than use post.jar directly. Can't we just include ant.jar and launch everything that way? I don't think this is a good idea. The problem is that you are building the command line by adding the args separated by spaces. Since Java 5 and even better with Java 8, you can use ProcessBuilder where you can add args and executable in a clean way. No need for the old-style ANT stuff (that is mainly there to allow running ANT on pre-Java-5).
          Hide
          Uwe Schindler added a comment -

          Thanks in any case! Works now!

          Show
          Uwe Schindler added a comment - Thanks in any case! Works now!
          Hide
          ASF subversion and git services added a comment -

          Commit 1695738 from Timothy Potter in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1695738 ]

          SOLR-7921: The techproducts example fails when running in a directory that contains spaces.

          Show
          ASF subversion and git services added a comment - Commit 1695738 from Timothy Potter in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1695738 ] SOLR-7921 : The techproducts example fails when running in a directory that contains spaces.
          Hide
          ASF subversion and git services added a comment -

          Commit 1695742 from Timothy Potter in branch 'dev/branches/lucene_solr_5_3'
          [ https://svn.apache.org/r1695742 ]

          SOLR-7921: The techproducts example fails when running in a directory that contains spaces.

          Show
          ASF subversion and git services added a comment - Commit 1695742 from Timothy Potter in branch 'dev/branches/lucene_solr_5_3' [ https://svn.apache.org/r1695742 ] SOLR-7921 : The techproducts example fails when running in a directory that contains spaces.
          Hide
          ASF subversion and git services added a comment -

          Commit 1695752 from Timothy Potter in branch 'dev/trunk'
          [ https://svn.apache.org/r1695752 ]

          SOLR-7921: Fix non-reproducible test failure on sarowe's Jenkins by adding a brief wait before checking that tech product docs have been committed to the example server

          Show
          ASF subversion and git services added a comment - Commit 1695752 from Timothy Potter in branch 'dev/trunk' [ https://svn.apache.org/r1695752 ] SOLR-7921 : Fix non-reproducible test failure on sarowe's Jenkins by adding a brief wait before checking that tech product docs have been committed to the example server
          Hide
          ASF subversion and git services added a comment -

          Commit 1695754 from Timothy Potter in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1695754 ]

          SOLR-7921: Fix non-reproducible test failure on sarowe's Jenkins by adding a brief wait before checking that tech product docs have been committed to the example server

          Show
          ASF subversion and git services added a comment - Commit 1695754 from Timothy Potter in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1695754 ] SOLR-7921 : Fix non-reproducible test failure on sarowe's Jenkins by adding a brief wait before checking that tech product docs have been committed to the example server
          Hide
          ASF subversion and git services added a comment -

          Commit 1695755 from Timothy Potter in branch 'dev/branches/lucene_solr_5_3'
          [ https://svn.apache.org/r1695755 ]

          SOLR-7921: Fix non-reproducible test failure on sarowe's Jenkins by adding a brief wait before checking that tech product docs have been committed to the example server

          Show
          ASF subversion and git services added a comment - Commit 1695755 from Timothy Potter in branch 'dev/branches/lucene_solr_5_3' [ https://svn.apache.org/r1695755 ] SOLR-7921 : Fix non-reproducible test failure on sarowe's Jenkins by adding a brief wait before checking that tech product docs have been committed to the example server
          Hide
          Noble Paul added a comment -

          I guess the test failures are related to this
          Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.3-Linux/60/
          Java: 64bit/jdk1.8.0_51 -XX:-UseCompressedOops -XX:+UseG1GC

          Show
          Noble Paul added a comment - I guess the test failures are related to this Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.3-Linux/60/ Java: 64bit/jdk1.8.0_51 -XX:-UseCompressedOops -XX:+UseG1GC
          Hide
          Uwe Schindler added a comment -

          I think the if (numFound == 0) should be a while (numFound == 0) loop. Sometimes Jenkins is very busy and threads are stalling.

          Show
          Uwe Schindler added a comment - I think the if (numFound == 0) should be a while (numFound == 0) loop. Sometimes Jenkins is very busy and threads are stalling.
          Hide
          Ishan Chattopadhyaya added a comment -

          So as not to hold up the release, I suggest we add the @Ignore back for the test (at least in the 5.3 branch) and continue to investigate in parallel (and perhaps try out the above suggestion, and retry for a few times).
          WDYT, Timothy Potter, Noble Paul, Uwe Schindler?

          Show
          Ishan Chattopadhyaya added a comment - So as not to hold up the release, I suggest we add the @Ignore back for the test (at least in the 5.3 branch) and continue to investigate in parallel (and perhaps try out the above suggestion, and retry for a few times). WDYT, Timothy Potter , Noble Paul , Uwe Schindler ?
          Hide
          Ishan Chattopadhyaya added a comment -

          Just to clarify and summarize: this issue (techproducts example not working with dirs containing spaces) was not present in 5.2. The issue, found in RC1, has been fixed here in this issue. The enabling of the test resulted in timing issues at Jenkins (works locally), and this is more of a test setup issue, as far as I can tell; the actual functionality seems to be working on both Windows and Linux.
          Hence, I suggest we ignore the test for now and go ahead with the release. (We can continue to investigate and fix the test later.)

          Show
          Ishan Chattopadhyaya added a comment - Just to clarify and summarize: this issue (techproducts example not working with dirs containing spaces) was not present in 5.2. The issue, found in RC1, has been fixed here in this issue. The enabling of the test resulted in timing issues at Jenkins (works locally), and this is more of a test setup issue, as far as I can tell; the actual functionality seems to be working on both Windows and Linux. Hence, I suggest we ignore the test for now and go ahead with the release. (We can continue to investigate and fix the test later.)
          Hide
          Noble Paul added a comment -

          can someone just confirm this and commit? i can respin the RC

          Show
          Noble Paul added a comment - can someone just confirm this and commit? i can respin the RC
          Hide
          ASF subversion and git services added a comment -

          Commit 1695930 from Timothy Potter in branch 'dev/trunk'
          [ https://svn.apache.org/r1695930 ]

          SOLR-7921: ignore testTechproductsExample unit test due to Jenkins failures

          Show
          ASF subversion and git services added a comment - Commit 1695930 from Timothy Potter in branch 'dev/trunk' [ https://svn.apache.org/r1695930 ] SOLR-7921 : ignore testTechproductsExample unit test due to Jenkins failures
          Hide
          ASF subversion and git services added a comment -

          Commit 1695931 from Timothy Potter in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1695931 ]

          SOLR-7921: ignore testTechproductsExample unit test due to Jenkins failures

          Show
          ASF subversion and git services added a comment - Commit 1695931 from Timothy Potter in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1695931 ] SOLR-7921 : ignore testTechproductsExample unit test due to Jenkins failures
          Hide
          ASF subversion and git services added a comment -

          Commit 1695932 from Timothy Potter in branch 'dev/branches/lucene_solr_5_3'
          [ https://svn.apache.org/r1695932 ]

          SOLR-7921: ignore testTechproductsExample unit test due to Jenkins failures

          Show
          ASF subversion and git services added a comment - Commit 1695932 from Timothy Potter in branch 'dev/branches/lucene_solr_5_3' [ https://svn.apache.org/r1695932 ] SOLR-7921 : ignore testTechproductsExample unit test due to Jenkins failures
          Hide
          Jan Høydahl added a comment -

          Is it best practice for this bug (and SOLR-7934) to have an entry under "Bug fixes" in CHANGES.TXT, when it fixes a never-released bug?

          Show
          Jan Høydahl added a comment - Is it best practice for this bug (and SOLR-7934 ) to have an entry under "Bug fixes" in CHANGES.TXT, when it fixes a never-released bug?
          Hide
          Erick Erickson added a comment -

          bq: ....when it fixes a never-released bug?

          I can argue either way, I don't think there's a hard and fast rule. I usually guess how many people are impacted by it and decide on that basis. IOW, "it's up to you".

          Show
          Erick Erickson added a comment - bq: ....when it fixes a never-released bug? I can argue either way, I don't think there's a hard and fast rule. I usually guess how many people are impacted by it and decide on that basis. IOW, "it's up to you".
          Hide
          Shalin Shekhar Mangar added a comment -

          I believe this was fixed in 5.3.0

          Show
          Shalin Shekhar Mangar added a comment - I believe this was fixed in 5.3.0

            People

            • Assignee:
              Timothy Potter
              Reporter:
              Ishan Chattopadhyaya
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development