Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7534

smokeTestRelease.py on cygwin [Errno 2] No such file or directory:

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0, 6.4
    • Component/s: None
    • Labels:
      None
    • Environment:

      Windows, Cygwin, Python

    • Lucene Fields:
      New

      Description

      $ python3 -u dev-tools/scripts/smokeTestRelease.py https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC2-rev1fe1a54db32b8c27bfae81887cd4d75242090613/
      Revision: 1fe1a54db32b8c27bfae81887cd4d75242090613
      Java 1.8 JAVA_HOME=C:\Program Files\Java\jdk1.8.0_102
      Traceback (most recent call last):
      File "dev-tools/scripts/smokeTestRelease.py", line 1440, in <module>
      main()
      File "dev-tools/scripts/smokeTestRelease.py", line 1377, in main
      c = parse_config()
      File "dev-tools/scripts/smokeTestRelease.py", line 1239, in parse_config
      c.java = make_java_config(parser, c.test_java8)
      File "dev-tools/scripts/smokeTestRelease.py", line 1193, in make_java_config
      run_java8 = _make_runner(java8_home, '1.8')
      File "dev-tools/scripts/smokeTestRelease.py", line 1179, in _make_runner
      java_home = subprocess.check_output('cygpath -u "%s"' % java_home).read().decode('utf-8').strip()
      File "/usr/lib/python3.4/subprocess.py", line 607, in check_output
      with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
      File "/usr/lib/python3.4/subprocess.py", line 859, in _init_
      restore_signals, start_new_session)
      File "/usr/lib/python3.4/subprocess.py", line 1457, in _execute_child
      raise child_exception_type(errno_num, err_msg)
      FileNotFoundError: [Errno 2] No such file or directory: 'cygpath -u "C:
      Program Files\\Java
      jdk1.8.0_102"'

      giving the doc path and args should be either supplied as array of terms or supplied as shell=True

      1. 6.3.0 RC3 smoke passed on cygwin.txt
        55 kB
        Mikhail Khludnev
      2. cloud test fail on smoke test.log
        1.23 MB
        Mikhail Khludnev
      3. fail due to long classpath test.log
        555 kB
        Mikhail Khludnev
      4. LUCENE-7534.patch
        4 kB
        Mikhail Khludnev
      5. LUCENE-7534.patch
        3 kB
        Mikhail Khludnev
      6. LUCENE-7534.patch
        3 kB
        Mikhail Khludnev
      7. LUCENE-7534.patch
        3 kB
        Mikhail Khludnev
      8. LUCENE-7534.patch
        3 kB
        Mikhail Khludnev
      9. LUCENE-7534.patch
        1 kB
        Mikhail Khludnev
      10. LUCENE-7534.patch
        1 kB
        Mikhail Khludnev
      11. LUCENE-7534.patch
        0.8 kB
        Mikhail Khludnev

        Issue Links

          Activity

          Hide
          mkhludnev Mikhail Khludnev added a comment -

          one more problem

          confirmAllReleasesAreTestedForBackCompat()
           stdout = stdout.decode('utf-8')
          UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf in position 17636: invalid continuation byte
          

          passing ,errors='ignore' seems fixes the problem.

          Show
          mkhludnev Mikhail Khludnev added a comment - one more problem confirmAllReleasesAreTestedForBackCompat() stdout = stdout.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf in position 17636: invalid continuation byte passing ,errors='ignore' seems fixes the problem.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          LUCENE-7534.patch let me finally smore release on widows (cygwin). Please, confirm that it won't hurt normal other platforms.

          Show
          mkhludnev Mikhail Khludnev added a comment - LUCENE-7534.patch let me finally smore release on widows (cygwin). Please, confirm that it won't hurt normal other platforms.
          Hide
          steve_rowe Steve Rowe added a comment -

          Thanks for working on it Mikhail - I put in the original cygwin support but haven't used windows much in years.

          I'll test your patch on Linux against the 6.3 RC.

          Show
          steve_rowe Steve Rowe added a comment - Thanks for working on it Mikhail - I put in the original cygwin support but haven't used windows much in years. I'll test your patch on Linux against the 6.3 RC.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          although..

          test solr example w/ Java 8...
          start Solr instance (log=/tmp/smoke_lucene_6.3.0_1fe1a54db32b8c27bfae81887cd4d75242090613_7/unpack/solr-6.3.0-java8/solr-example.log)...
          This script does not support cygwin due to severe limitations and lack of adherence
          to BASH standards, such as lack of lsof, curl, and ps options.

          Please use the native solr.cmd script on Windows!
          Running techproducts example on port 8983 from /tmp/smoke_lucene_6.3.0_1fe1a54db32b8c27bfae81887cd4d75242090613_7/unpack/solr-6.3.0-java8
          This script does not support cygwin due to severe limitations and lack of adherence
          to BASH standards, such as lack of lsof, curl, and ps options.

          Please use the native solr.cmd script on Windows!
          stop server using: bin/solr stop -p 8983
          This script does not support cygwin due to severe limitations and lack of adherence
          to BASH standards, such as lack of lsof, curl, and ps options.

          Please use the native solr.cmd script

          Show
          mkhludnev Mikhail Khludnev added a comment - although.. test solr example w/ Java 8... start Solr instance (log=/tmp/smoke_lucene_6.3.0_1fe1a54db32b8c27bfae81887cd4d75242090613_7/unpack/solr-6.3.0-java8/solr-example.log)... This script does not support cygwin due to severe limitations and lack of adherence to BASH standards, such as lack of lsof, curl, and ps options. Please use the native solr.cmd script on Windows! Running techproducts example on port 8983 from /tmp/smoke_lucene_6.3.0_1fe1a54db32b8c27bfae81887cd4d75242090613_7/unpack/solr-6.3.0-java8 This script does not support cygwin due to severe limitations and lack of adherence to BASH standards, such as lack of lsof, curl, and ps options. Please use the native solr.cmd script on Windows! stop server using: bin/solr stop -p 8983 This script does not support cygwin due to severe limitations and lack of adherence to BASH standards, such as lack of lsof, curl, and ps options. Please use the native solr.cmd script
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I suppose it's worth to tweak it to call solr.cmd on cygwin, yet more work need to be done here...

          Show
          mkhludnev Mikhail Khludnev added a comment - I suppose it's worth to tweak it to call solr.cmd on cygwin, yet more work need to be done here...
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          ok. I changed smokeTestRelease.py to invoke solr.cmd, the fail is following

          $ ./solr/bin/solr.cmd stop -p 8983
          find: ‘TCP ’: No such file or directory
          find: ‘:8983 ’: No such file or directory
          find: ‘:0 ’: No such file or directory
          No Solr found running on port 8983

          Show
          mkhludnev Mikhail Khludnev added a comment - ok. I changed smokeTestRelease.py to invoke solr.cmd, the fail is following $ ./solr/bin/solr.cmd stop -p 8983 find: ‘TCP ’: No such file or directory find: ‘:8983 ’: No such file or directory find: ‘:0 ’: No such file or directory No Solr found running on port 8983
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          cygwin's find messed windows' find up

          Show
          mkhludnev Mikhail Khludnev added a comment - cygwin's find messed windows' find up
          Hide
          mkhludnev Mikhail Khludnev added a comment - - edited

          Here is the status so far: I launch smokeTestRelease.py from cygwin, thus when it invokes bin/solr.cmd it got cygwin's PATH, where cygwin's find (yep, it can be replaced to findstr) and timeout, and perhaps something else hijacks calls from same named windows programs. And here I really have no idea what to do with it. Is there any ideas?

          Show
          mkhludnev Mikhail Khludnev added a comment - - edited Here is the status so far: I launch smokeTestRelease.py from cygwin, thus when it invokes bin/solr.cmd it got cygwin's PATH, where cygwin's find (yep, it can be replaced to findstr ) and timeout , and perhaps something else hijacks calls from same named windows programs. And here I really have no idea what to do with it. Is there any ideas?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          however

          env "PATH=`cygpath -S -w`:$PATH" timeout

          calls windows timeout, indeed!!

          Show
          mkhludnev Mikhail Khludnev added a comment - however env "PATH=`cygpath -S -w`:$PATH" timeout calls windows timeout, indeed!!
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I've got this with the last patch

          test solr example w/ Java 8...
          start Solr instance (log=/tmp/smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4/unpack/solr-6.3.0-java8/solr-example.log)...
          No Solr found running on port 8983
          Running techproducts example on port 8983 from /tmp/smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4/unpack/solr-6.3.0-java8
          Creating Solr home directory C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\example\techproducts\solr

          Starting up Solr on port 8983 using command:
          C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\bin\solr.cmd start -p 8983 -s "C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\example\techproducts\solr"

          Waiting up to 30 to see Solr running on port 8983
          Started Solr server on port 8983. Happy searching!

          Copying configuration to new core instance directory:
          C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\example\techproducts\solr\techproducts

          Creating new core 'techproducts' using command:
          http://localhost:8983/solr/admin/cores?action=CREATE&name=techproducts&instanceDir=techproducts

          {
          "responseHeader":

          Unknown macro: { "status"}

          ,
          "core":"techproducts"}

          Indexing tech product example docs from C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\example\exampledocs
          SimplePostTool version 5.0.0
          Posting files to [base] url http://localhost:8983/solr/techproducts/update using content-type application/xml...
          POSTing file gb18030-example.xml to [base]
          POSTing file hd.xml to [base]
          POSTing file ipod_other.xml to [base]
          POSTing file ipod_video.xml to [base]
          POSTing file manufacturers.xml to [base]
          POSTing file mem.xml to [base]
          POSTing file money.xml to [base]
          POSTing file monitor.xml to [base]
          POSTing file monitor2.xml to [base]
          POSTing file mp500.xml to [base]
          POSTing file sd500.xml to [base]
          POSTing file solr.xml to [base]
          POSTing file utf8-example.xml to [base]
          POSTing file vidcard.xml to [base]
          14 files indexed.
          COMMITting Solr index changes to http://localhost:8983/solr/techproducts/update...
          Time spent: 0:00:00.609

          Solr techproducts example launched successfully. Direct your Web browser to http://localhost:8983/solr to visit the Solr Admin UI
          test utf8...
          run query...
          stop server using: bin/solr stop -p 8983
          Stopping Solr process 7956 running on port 8983
          ERROR: Input redirection is not supported, exiting the process immediately.

          So, it works, so far, I have no idea about the ERROR in the last line.

          Show
          mkhludnev Mikhail Khludnev added a comment - I've got this with the last patch test solr example w/ Java 8... start Solr instance (log=/tmp/smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4/unpack/solr-6.3.0-java8/solr-example.log)... No Solr found running on port 8983 Running techproducts example on port 8983 from /tmp/smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4/unpack/solr-6.3.0-java8 Creating Solr home directory C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\example\techproducts\solr Starting up Solr on port 8983 using command: C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\bin\solr.cmd start -p 8983 -s "C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\example\techproducts\solr" Waiting up to 30 to see Solr running on port 8983 Started Solr server on port 8983. Happy searching! Copying configuration to new core instance directory: C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\example\techproducts\solr\techproducts Creating new core 'techproducts' using command: http://localhost:8983/solr/admin/cores?action=CREATE&name=techproducts&instanceDir=techproducts { "responseHeader": Unknown macro: { "status"} , "core":"techproducts"} Indexing tech product example docs from C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0-java8\example\exampledocs SimplePostTool version 5.0.0 Posting files to [base] url http://localhost:8983/solr/techproducts/update using content-type application/xml... POSTing file gb18030-example.xml to [base] POSTing file hd.xml to [base] POSTing file ipod_other.xml to [base] POSTing file ipod_video.xml to [base] POSTing file manufacturers.xml to [base] POSTing file mem.xml to [base] POSTing file money.xml to [base] POSTing file monitor.xml to [base] POSTing file monitor2.xml to [base] POSTing file mp500.xml to [base] POSTing file sd500.xml to [base] POSTing file solr.xml to [base] POSTing file utf8-example.xml to [base] POSTing file vidcard.xml to [base] 14 files indexed. COMMITting Solr index changes to http://localhost:8983/solr/techproducts/update ... Time spent: 0:00:00.609 Solr techproducts example launched successfully. Direct your Web browser to http://localhost:8983/solr to visit the Solr Admin UI test utf8... run query... stop server using: bin/solr stop -p 8983 Stopping Solr process 7956 running on port 8983 ERROR: Input redirection is not supported, exiting the process immediately. So, it works, so far, I have no idea about the ERROR in the last line.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          ok. it's caused by windows' timeout in solr.cmd, here is the proof

          $ python3
          Python 3.4.3 (default, May 5 2015, 17:58:45)
          [GCC 4.9.2] on cygwin
          Type "help", "copyright", "credits" or "license" for more information.
          >>> import subprocess
          >>> subprocess.call('env "PATH=`cygpath -S -w`:$PATH" timeout /T 5', shell=True)
          ERROR: Input redirection is not supported, exiting the process immediately.
          1

          I don't think it's a problem.

          Show
          mkhludnev Mikhail Khludnev added a comment - ok. it's caused by windows' timeout in solr.cmd , here is the proof $ python3 Python 3.4.3 (default, May 5 2015, 17:58:45) [GCC 4.9.2] on cygwin Type "help", "copyright", "credits" or "license" for more information. >>> import subprocess >>> subprocess.call('env "PATH=`cygpath -S -w`:$PATH" timeout /T 5', shell=True) ERROR: Input redirection is not supported, exiting the process immediately. 1 I don't think it's a problem.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Dead end. Randomized test runner and spawn a slave jvm because its' classpath exceeds command line length limit. It occurs in contrib\solr-map-reduce. see fail due to long classpath test.log

          [junit4] Caused by: java.io.IOException: Cannot run program "C:\Program Files\Java\jdk1.8.0_102\jre\bin\java.exe" (in directory "C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\contrib\solr-map-reduce\test\J2"): CreateProcess error=206, The filename or extension is too long

          Show
          mkhludnev Mikhail Khludnev added a comment - Dead end. Randomized test runner and spawn a slave jvm because its' classpath exceeds command line length limit. It occurs in contrib\solr-map-reduce . see fail due to long classpath test.log [junit4] Caused by: java.io.IOException: Cannot run program "C:\Program Files\Java\jdk1.8.0_102\jre\bin\java.exe" (in directory "C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\contrib\solr-map-reduce\test\J2"): CreateProcess error=206, The filename or extension is too long
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Dawid Weiss, is there a way to shorten slave command line and/or classpath? Can't it use jar folders, or switch paths to relative ones?

          [junit4] Caused by: java.io.IOException: CreateProcess error=206, The filename or extension is too long
          [junit4] at java.lang.ProcessImpl.create(Native Method)
          [junit4] at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)
          [junit4] at java.lang.ProcessImpl.start(ProcessImpl.java:137)
          [junit4] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
          [junit4] ... 13 more
          [junit4] ERROR: JVM J2 ended with an exception, command line: "C:\Program Files\Java\jdk1.8.0_102\jre\bin\java.exe" -ea -esa -Dtests.prefix=tests -Dtests.seed=5AA2B74D3487DDFB -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=6.3.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\tools\junit4\logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=false -Dtests.asserts=true -Dtests.multiplier=1 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\contrib\solr-map-reduce\test\temp -Dcommon.dir=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene -Dclover.db.dir=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\build\clover\db -Djava.security.policy=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\tools\junit4\solr-tests.policy -Dtests.LUCENE_VERSION=6.3.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Djunit4.childvm.cwd=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\contrib\solr-map-reduce\test\J2 -Djunit4.childvm.id=2 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Dtests.disableHdfs=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath "C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\contrib\solr-map-reduce\classes\test;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\solr-test-framework\classes\java;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\test-framework\lib\junit4-ant-2.4.0.jar;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\contrib\map-reduce\src\test-files;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\build\test-framework\classes\java;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\build\codecs\classes\java;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\solr-solrj\classes\java;C:\cygwin64..

          {qoute}
          Show
          mkhludnev Mikhail Khludnev added a comment - Dawid Weiss , is there a way to shorten slave command line and/or classpath? Can't it use jar folders, or switch paths to relative ones? [junit4] Caused by: java.io.IOException: CreateProcess error=206, The filename or extension is too long [junit4] at java.lang.ProcessImpl.create(Native Method) [junit4] at java.lang.ProcessImpl.<init>(ProcessImpl.java:386) [junit4] at java.lang.ProcessImpl.start(ProcessImpl.java:137) [junit4] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) [junit4] ... 13 more [junit4] ERROR: JVM J2 ended with an exception, command line: "C:\Program Files\Java\jdk1.8.0_102\jre\bin\java.exe" -ea -esa -Dtests.prefix=tests -Dtests.seed=5AA2B74D3487DDFB -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=6.3.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\tools\junit4\logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=false -Dtests.asserts=true -Dtests.multiplier=1 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\contrib\solr-map-reduce\test\temp -Dcommon.dir=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene -Dclover.db.dir=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\build\clover\db -Djava.security.policy=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\tools\junit4\solr-tests.policy -Dtests.LUCENE_VERSION=6.3.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Djunit4.childvm.cwd=C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\contrib\solr-map-reduce\test\J2 -Djunit4.childvm.id=2 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Dtests.disableHdfs=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath "C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\contrib\solr-map-reduce\classes\test;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\solr-test-framework\classes\java;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\test-framework\lib\junit4-ant-2.4.0.jar;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\contrib\map-reduce\src\test-files;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\build\test-framework\classes\java;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\lucene\build\codecs\classes\java;C:\cygwin64\tmp\smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4\unpack\solr-6.3.0\solr\build\solr-solrj\classes\java;C:\cygwin64.. {qoute}
          Hide
          dweiss Dawid Weiss added a comment -

          Paths added via -D are passed verbatim from ant's buildfile, the runner has nothing to do with it (so are classpath entries, actually).

          You could modify it to use relative paths, but verbosity here will save you days in debugging should something go wrong and there are classpath lookup issues somewhere... Absolute classpaths also permit low-level rerunning of the same "java" command used to fork the process, which again is very handy for debugging.

          smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4

          You could change this part to use short git rev (again – this is in the smoke tester/ build files).

          Can't it use jar folders, or switch paths to relative ones?

          What are JAR folders? If you mean overriding ext dirs property, then no, it's different class loader and runtime behavior. If you mean a synthetic JAR with manifest entries then, again, no – this would be different from a vanilla --cp run and the runner shouldn't alter the default behavior here.

          Show
          dweiss Dawid Weiss added a comment - Paths added via -D are passed verbatim from ant's buildfile, the runner has nothing to do with it (so are classpath entries, actually). You could modify it to use relative paths, but verbosity here will save you days in debugging should something go wrong and there are classpath lookup issues somewhere... Absolute classpaths also permit low-level rerunning of the same "java" command used to fork the process, which again is very handy for debugging. smoke_lucene_6.3.0_a66a44513ee8191e25b477372094bfa846450316_4 You could change this part to use short git rev (again – this is in the smoke tester/ build files). Can't it use jar folders, or switch paths to relative ones? What are JAR folders? If you mean overriding ext dirs property, then no, it's different class loader and runtime behavior. If you mean a synthetic JAR with manifest entries then, again, no – this would be different from a vanilla --cp run and the runner shouldn't alter the default behavior here.
          Hide
          dweiss Dawid Weiss added a comment -

          Btw. there is a way to pass arguments to "java" bootstrap via an external file, but this is only available from 9+. See this JEP:
          http://openjdk.java.net/jeps/293

          Show
          dweiss Dawid Weiss added a comment - Btw. there is a way to pass arguments to "java" bootstrap via an external file, but this is only available from 9+. See this JEP: http://openjdk.java.net/jeps/293
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I respin it again passing a short --tmp-dir, obviously... sorry for bothering you.

          Show
          mkhludnev Mikhail Khludnev added a comment - I respin it again passing a short --tmp-dir , obviously... sorry for bothering you.
          Hide
          dweiss Dawid Weiss added a comment -

          No problem at all.

          Show
          dweiss Dawid Weiss added a comment - No problem at all.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          ok. it helped a little. Now it works like:

          .. everything is fine
            unpack solr-6.3.0.tgz...
          Starting up Solr on port 8983 using command:
          C:\cygwin64\tmp\rc3\unpack\solr-6.3.0-java8\bin\solr.cmd start -p 8983 -s "C:\cygwin64\tmp\rc3\unpack\solr-6.3.0-java8\example\techproducts\solr"
          
          Waiting up to 30 to see Solr running on port 8983
          Started Solr server on port 8983. Happy searching!
          ...
            unpack solr-6.3.0.zip...
          C:\cygwin64\tmp\rc3\unpack\solr-6.3.0-java8\bin\solr.cmd start -p 8983 -s "C:\cygwin64\tmp\rc3\unpack\solr-6.3.0-java8\example\techproducts\solr"
          
          Waiting up to 30 to see Solr running on port 8983
          Started Solr server on port 8983. Happy searching!
          ...
            unpack solr-6.3.0-src.tgz...
          ...
          Verify...
              test solr example w/ Java 8...
                start Solr instance (log=/tmp/rc3/unpack/solr-6.3.0/solr-example.log)...
          env: ‘bin/solr.cmd’: Permission denied
                Running techproducts example on port 8983 from /tmp/rc3/unpack/solr-6.3.0
          env: ‘bin/solr.cmd’: Permission denied
                stop server using: bin/solr stop -p 8983
          env: ‘bin/solr.cmd’: Permission denied
          Traceback (most recent call last):
            File "dev-tools/scripts/smokeTestRelease.py", line 1447, in <module>
              main()
            File "dev-tools/scripts/smokeTestRelease.py", line 1391, in main
              smokeTest(c.java, c.url, c.revision, c.version, c.tmp_dir, c.is_signed, ' '.join(c.test_args))
            File "dev-tools/scripts/smokeTestRelease.py", line 1437, in smokeTest
              gitRevision, version, testArgs, baseURL)
            File "dev-tools/scripts/smokeTestRelease.py", line 597, in unpackAndVerify
              verifyUnpacked(java, project, artifact, unpackPath, gitRevision, version, testArgs, tmpDir, baseURL)
            File "dev-tools/scripts/smokeTestRelease.py", line 711, in verifyUnpacked
              testSolrExample(unpackPath, java.java8_home, True)
            File "dev-tools/scripts/smokeTestRelease.py", line 823, in testSolrExample
              raise RuntimeError('Failed to run the techproducts example, check log for previous errors.')
          RuntimeError: Failed to run the techproducts example, check log for previous errors.
          
          

          it seems like solr-6.3.0-src.tgz doesn't carry executable attr for solr.cmd, at contrast to solr-6.3.0.zip, solr-6.3.0.tgz. What's the trick over there? It should hardly be different for windows? or..

          Show
          mkhludnev Mikhail Khludnev added a comment - ok. it helped a little. Now it works like: .. everything is fine unpack solr-6.3.0.tgz... Starting up Solr on port 8983 using command: C:\cygwin64\tmp\rc3\unpack\solr-6.3.0-java8\bin\solr.cmd start -p 8983 -s "C:\cygwin64\tmp\rc3\unpack\solr-6.3.0-java8\example\techproducts\solr" Waiting up to 30 to see Solr running on port 8983 Started Solr server on port 8983. Happy searching! ... unpack solr-6.3.0.zip... C:\cygwin64\tmp\rc3\unpack\solr-6.3.0-java8\bin\solr.cmd start -p 8983 -s "C:\cygwin64\tmp\rc3\unpack\solr-6.3.0-java8\example\techproducts\solr" Waiting up to 30 to see Solr running on port 8983 Started Solr server on port 8983. Happy searching! ... unpack solr-6.3.0-src.tgz... ... Verify... test solr example w/ Java 8... start Solr instance (log=/tmp/rc3/unpack/solr-6.3.0/solr-example.log)... env: ‘bin/solr.cmd’: Permission denied Running techproducts example on port 8983 from /tmp/rc3/unpack/solr-6.3.0 env: ‘bin/solr.cmd’: Permission denied stop server using: bin/solr stop -p 8983 env: ‘bin/solr.cmd’: Permission denied Traceback (most recent call last): File "dev-tools/scripts/smokeTestRelease.py" , line 1447, in <module> main() File "dev-tools/scripts/smokeTestRelease.py" , line 1391, in main smokeTest(c.java, c.url, c.revision, c.version, c.tmp_dir, c.is_signed, ' '.join(c.test_args)) File "dev-tools/scripts/smokeTestRelease.py" , line 1437, in smokeTest gitRevision, version, testArgs, baseURL) File "dev-tools/scripts/smokeTestRelease.py" , line 597, in unpackAndVerify verifyUnpacked(java, project, artifact, unpackPath, gitRevision, version, testArgs, tmpDir, baseURL) File "dev-tools/scripts/smokeTestRelease.py" , line 711, in verifyUnpacked testSolrExample(unpackPath, java.java8_home, True) File "dev-tools/scripts/smokeTestRelease.py" , line 823, in testSolrExample raise RuntimeError('Failed to run the techproducts example, check log for previous errors.') RuntimeError: Failed to run the techproducts example, check log for previous errors. it seems like solr-6.3.0-src.tgz doesn't carry executable attr for solr.cmd, at contrast to solr-6.3.0.zip, solr-6.3.0.tgz. What's the trick over there? It should hardly be different for windows? or..
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I'm sure I almost done with it. but releases have been disappeared. Is there an url to practice on? or how to get RC locally and point on it via file:// url ?

          Show
          mkhludnev Mikhail Khludnev added a comment - I'm sure I almost done with it. but releases have been disappeared. Is there an url to practice on? or how to get RC locally and point on it via file:// url ?
          Hide
          steve_rowe Steve Rowe added a comment -

          You can retrieve deleted revisions like this:

          svn export --revision=16808 https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC3-reva66a44513ee8191e25b477372094bfa846450316/
          

          and then tell the smoke tester to use the local files like this:

          python3 -u dev-tools/scripts/smokeTestRelease.py file:///Users/sarowe/temp/lucene-solr-6.3.0-RC3-reva66a44513ee8191e25b477372094bfa846450316/
          

          Looks like your Python is from Cygwin, so you should be able to use a file:///home/username/path/to/export style URL.

          Show
          steve_rowe Steve Rowe added a comment - You can retrieve deleted revisions like this: svn export --revision=16808 https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC3-reva66a44513ee8191e25b477372094bfa846450316/ and then tell the smoke tester to use the local files like this: python3 -u dev-tools/scripts/smokeTestRelease.py file:///Users/sarowe/temp/lucene-solr-6.3.0-RC3-reva66a44513ee8191e25b477372094bfa846450316/ Looks like your Python is from Cygwin, so you should be able to use a file:///home/username/path/to/export style URL.
          Hide
          steve_rowe Steve Rowe added a comment -

          This is a direct link that the Subversion server understands to the deleted 6.3.0 RC3 revision: https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC3-reva66a44513ee8191e25b477372094bfa846450316?p=16808, but the smoke tester's link normalization code seems to strip off the revision, so it doesn't work.

          Show
          steve_rowe Steve Rowe added a comment - This is a direct link that the Subversion server understands to the deleted 6.3.0 RC3 revision: https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC3-reva66a44513ee8191e25b477372094bfa846450316?p=16808 , but the smoke tester's link normalization code seems to strip off the revision, so it doesn't work.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          turns out it can be pulled with svn co -r 16808 http://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC3-reva66a44513ee8191e25b477372094bfa846450316 I gonna make the next atempt.

          Show
          mkhludnev Mikhail Khludnev added a comment - turns out it can be pulled with svn co -r 16808 http://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC3-reva66a44513ee8191e25b477372094bfa846450316 I gonna make the next atempt.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I noticed that tests are rather slow, checking jstack gets:

          "SUITE-TestCrossCoreJoin-seed#[CBDD0D910C629184]-worker" #1395 prio=5 os_prio=0 tid=0x000000001eeb3800 nid=0x4284 runnable [0x00000000718bd000]
             java.lang.Thread.State: RUNNABLE
                  at java.net.Inet6AddressImpl.getHostByAddr(Native Method)
                  at java.net.InetAddress$2.getHostByAddr(InetAddress.java:932)
                  at java.net.InetAddress.getHostFromNameService(InetAddress.java:617)
                  at java.net.InetAddress.getHostName(InetAddress.java:559)
                  at java.net.SocketPermission.getCanonName(SocketPermission.java:666)
                  at java.net.SocketPermission.impliesIgnoreMask(SocketPermission.java:968)
                  at java.net.SocketPermissionCollection.implies(SocketPermission.java:1402)
                  - locked <0x00000000e00244e0> (a java.net.SocketPermissionCollection)
                  at java.security.Permissions.implies(Permissions.java:182)
                  - locked <0x00000000e0023d20> (a java.security.Permissions)
                  at sun.security.provider.PolicyFile.implies(PolicyFile.java:1083)
                  at java.security.ProtectionDomain.implies(ProtectionDomain.java:281)
                  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:450)
                  at java.security.AccessController.checkPermission(AccessController.java:884)
                  at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
                  at java.lang.SecurityManager.checkConnect(SecurityManager.java:1048)
                  at java.net.InetAddress.getAllByName0(InetAddress.java:1268)
                  at java.net.InetAddress.getAllByName0(InetAddress.java:1253)
                  at java.net.InetAddress.getHostFromNameService(InetAddress.java:634)
                  at java.net.InetAddress.getCanonicalHostName(InetAddress.java:588)
                  at org.apache.solr.handler.admin.SystemInfoHandler.init(SystemInfoHandler.java:91)
                  at org.apache.solr.handler.admin.SystemInfoHandler.<init>(SystemInfoHandler.java:85)
                  at org.apache.solr.handler.admin.InfoHandler.<init>(InfoHandler.java:45)
                  at sun.reflect.GeneratedConstructorAccessor125.newInstance(Unknown Source)
                  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                  at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:637)
          

          If there is no other ideas I'll pass -Djava.net.preferIPv4Stack=true next time.

          Show
          mkhludnev Mikhail Khludnev added a comment - I noticed that tests are rather slow, checking jstack gets: "SUITE-TestCrossCoreJoin-seed#[CBDD0D910C629184]-worker" #1395 prio=5 os_prio=0 tid=0x000000001eeb3800 nid=0x4284 runnable [0x00000000718bd000] java.lang. Thread .State: RUNNABLE at java.net.Inet6AddressImpl.getHostByAddr(Native Method) at java.net.InetAddress$2.getHostByAddr(InetAddress.java:932) at java.net.InetAddress.getHostFromNameService(InetAddress.java:617) at java.net.InetAddress.getHostName(InetAddress.java:559) at java.net.SocketPermission.getCanonName(SocketPermission.java:666) at java.net.SocketPermission.impliesIgnoreMask(SocketPermission.java:968) at java.net.SocketPermissionCollection.implies(SocketPermission.java:1402) - locked <0x00000000e00244e0> (a java.net.SocketPermissionCollection) at java.security.Permissions.implies(Permissions.java:182) - locked <0x00000000e0023d20> (a java.security.Permissions) at sun.security.provider.PolicyFile.implies(PolicyFile.java:1083) at java.security.ProtectionDomain.implies(ProtectionDomain.java:281) at java.security.AccessControlContext.checkPermission(AccessControlContext.java:450) at java.security.AccessController.checkPermission(AccessController.java:884) at java.lang. SecurityManager .checkPermission( SecurityManager .java:549) at java.lang. SecurityManager .checkConnect( SecurityManager .java:1048) at java.net.InetAddress.getAllByName0(InetAddress.java:1268) at java.net.InetAddress.getAllByName0(InetAddress.java:1253) at java.net.InetAddress.getHostFromNameService(InetAddress.java:634) at java.net.InetAddress.getCanonicalHostName(InetAddress.java:588) at org.apache.solr.handler.admin.SystemInfoHandler.init(SystemInfoHandler.java:91) at org.apache.solr.handler.admin.SystemInfoHandler.<init>(SystemInfoHandler.java:85) at org.apache.solr.handler.admin.InfoHandler.<init>(InfoHandler.java:45) at sun.reflect.GeneratedConstructorAccessor125.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:637) If there is no other ideas I'll pass -Djava.net.preferIPv4Stack=true next time.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          solr cluster tests can't pass

          Show
          mkhludnev Mikhail Khludnev added a comment - solr cluster tests can't pass
          Hide
          mkhludnev Mikhail Khludnev added a comment - - edited

          ok. solr test passed. I've got an old new PITA. cygwin x permission strike back again when solr.cmd tries to spawn solr with example (sadly SolrCli swallow the problem message , see below). Only solr-..src.tgz ?? suffer from it so far. -What's an idea? can archive bring x premission or smoke script should chmod it?

            unpack solr-6.3.0-src.tgz...
          .....
          Starting up Solr on port 8983 using command:
          C:\cygwin64\tmp\s\unpack\solr-6.3.0\solr\bin\solr.cmd start -p 8983 -s "C:\cygwin64\tmp\s\unpack\solr-6.3.0\solr\example\techproducts\solr"
          
          -559038737 org.apache.commons.exec.ExecuteException: Execution failed (Exit value: -559038737. Caused by java.io.IOException: Cannot run program "C:\cygwin64\tmp\s\unpack\solr-6.3.0\solr\bin\solr.cmd" (in directory "."): CreateProcess error=5, Access is denied)
          Checking status of Solr at http://localhost:8983/solr ...
          
          ERROR: Did not see Solr at http://localhost:8983/solr come online within 30
          
          
          Show
          mkhludnev Mikhail Khludnev added a comment - - edited ok. solr test passed. I've got an old new PITA. cygwin x permission strike back again when solr.cmd tries to spawn solr with example (sadly SolrCli swallow the problem message , see below). Only solr-.. src.tgz ?? suffer from it so far. -What's an idea? can archive bring x premission or smoke script should chmod it? unpack solr-6.3.0-src.tgz... ..... Starting up Solr on port 8983 using command: C:\cygwin64\tmp\s\unpack\solr-6.3.0\solr\bin\solr.cmd start -p 8983 -s "C:\cygwin64\tmp\s\unpack\solr-6.3.0\solr\example\techproducts\solr" -559038737 org.apache.commons.exec.ExecuteException: Execution failed (Exit value: -559038737. Caused by java.io.IOException: Cannot run program "C:\cygwin64\tmp\s\unpack\solr-6.3.0\solr\bin\solr.cmd" (in directory "." ): CreateProcess error=5, Access is denied) Checking status of Solr at http: //localhost:8983/solr ... ERROR: Did not see Solr at http: //localhost:8983/solr come online within 30
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I've got it working after all. 6.3.0 RC3 smoke passed on cygwin.txt
          Take a pleasure to review LUCENE-7534.patch, it would be great to check these change on linux on downloaded RC. I'm really keen (after all this pain) to commit it until next release.

          Show
          mkhludnev Mikhail Khludnev added a comment - I've got it working after all. 6.3.0 RC3 smoke passed on cygwin.txt Take a pleasure to review LUCENE-7534.patch , it would be great to check these change on linux on downloaded RC. I'm really keen (after all this pain) to commit it until next release.
          Hide
          steve_rowe Steve Rowe added a comment -

          Patch looks good (didn't actually run it on Cygwin though).

          I successfully ran the smoke tester with your latest patch on Linux against a local 6.3.0 RC3: SUCCESS! [0:24:03.418973]

          +1 to commit.

          Show
          steve_rowe Steve Rowe added a comment - Patch looks good (didn't actually run it on Cygwin though). I successfully ran the smoke tester with your latest patch on Linux against a local 6.3.0 RC3: SUCCESS! [0:24:03.418973] +1 to commit.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a209c4e8c259c66b8748656207d36db623d50856 in lucene-solr's branch refs/heads/master from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a209c4e ]

          LUCENE-7534: fix smokeTestRelease.py to work on Cygwin

          Show
          jira-bot ASF subversion and git services added a comment - Commit a209c4e8c259c66b8748656207d36db623d50856 in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a209c4e ] LUCENE-7534 : fix smokeTestRelease.py to work on Cygwin
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit fb4abd605de93b4e7283ad5a172e87a67c15f223 in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fb4abd6 ]

          LUCENE-7534: fix smokeTestRelease.py to work on Cygwin

          Show
          jira-bot ASF subversion and git services added a comment - Commit fb4abd605de93b4e7283ad5a172e87a67c15f223 in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fb4abd6 ] LUCENE-7534 : fix smokeTestRelease.py to work on Cygwin

            People

            • Assignee:
              mkhludnev Mikhail Khludnev
              Reporter:
              mkhludnev Mikhail Khludnev
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development