Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8689

bin/solr.cmd does not start with recent Verona builds of Java 9 because of version parsing issue

    Details

      Description

      At least on Windows, Solr 5.5 does not start with the shell script using a Verona-Java-9 JDK:

      *****************************************************
      JAVA_HOME = C:\Program Files\Java\jdk-9
      java version "9-ea"
      Java(TM) SE Runtime Environment (build 9-ea+105-2016-02-11-003336.javare.4433.nc)
      Java HotSpot(TM) 64-Bit Server VM (build 9-ea+105-2016-02-11-003336.javare.4433.nc, mixed mode)
      *****************************************************
      C:\Users\Uwe Schindler\Desktop\solr-5.5.0\bin>solr start
      ERROR: Java 1.7 or later is required to run Solr. Current Java version is: 9-ea
      

      I don't know if this is better with Linux, but I assume the version parsing is broken (e.g., String#startsWith, interpret as floating point number,...)

      We should fix this before Java 9 gets released! The version numbering scheme changed completely: http://openjdk.java.net/jeps/223

      1. SOLR-8689.patch
        5 kB
        Uwe Schindler
      2. SOLR-8689.patch
        4 kB
        Uwe Schindler
      3. SOLR-8689.patch
        4 kB
        Uwe Schindler
      4. SOLR-8689.patch
        4 kB
        Uwe Schindler
      5. SOLR-8689.patch
        4 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          hossman Hoss Man added a comment -

          I don't have a windows VM to use to work on this, but whoever picks it up might want to review the corollary fixes in bin/solr over in SOLR-10184: in particular this comment...

          https://issues.apache.org/jira/browse/SOLR-10184?focusedCommentId=15904091&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15904091

          • fix the version parsing to look at the entire number, and compare against entire float versions like "1.8"
          • fix gc log option defaults & how log file is added

          bash changes that should be mimiced in windows cmd scripting...
          https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=09bd861

          Show
          hossman Hoss Man added a comment - I don't have a windows VM to use to work on this, but whoever picks it up might want to review the corollary fixes in bin/solr over in SOLR-10184 : in particular this comment... https://issues.apache.org/jira/browse/SOLR-10184?focusedCommentId=15904091&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15904091 fix the version parsing to look at the entire number, and compare against entire float versions like "1.8" fix gc log option defaults & how log file is added bash changes that should be mimiced in windows cmd scripting... https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=09bd861
          Hide
          thetaphi Uwe Schindler added a comment -

          Hi Hoss,

          I will quickly try to start it with Java 9 on my Windows machine.

          I plan to enable Java 9 EA builds on Policeman Jenkins Windows Slave, so we get better testing, too. Currently we only run Java 9 on Linux (mostly because of maintenance, which is hard with GUI based Windows Installers instead of automated unzipping of tar.gz and renaming install dirs...).

          Show
          thetaphi Uwe Schindler added a comment - Hi Hoss, I will quickly try to start it with Java 9 on my Windows machine. I plan to enable Java 9 EA builds on Policeman Jenkins Windows Slave, so we get better testing, too. Currently we only run Java 9 on Linux (mostly because of maintenance, which is hard with GUI based Windows Installers instead of automated unzipping of tar.gz and renaming install dirs...).
          Hide
          thetaphi Uwe Schindler added a comment -

          Current state with master:

          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>java -version
          java version "9-ea"
          Java(TM) SE Runtime Environment (build 9-ea+159)
          Java HotSpot(TM) 64-Bit Server VM (build 9-ea+159, mixed mode)
          
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr start
          
          ERROR: Java 1.8 or later is required to run Solr. Current Java version is: 9-ea
          
          Show
          thetaphi Uwe Schindler added a comment - Current state with master: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>java -version java version "9-ea" Java(TM) SE Runtime Environment (build 9-ea+159) Java HotSpot(TM) 64-Bit Server VM (build 9-ea+159, mixed mode) C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr start ERROR: Java 1.8 or later is required to run Solr. Current Java version is: 9-ea
          Hide
          janhoy Jan Høydahl added a comment -

          Important: This issue should also take care of the GC settings which were done for bin/solr in SOLR-10184

          Show
          janhoy Jan Høydahl added a comment - Important: This issue should also take care of the GC settings which were done for bin/solr in SOLR-10184
          Hide
          thetaphi Uwe Schindler added a comment -

          I am fixing this issue right now. I got the version parsing working. I will now add the GC settings (as Solr does not start because of the GC options).

          Show
          thetaphi Uwe Schindler added a comment - I am fixing this issue right now. I got the version parsing working. I will now add the GC settings (as Solr does not start because of the GC options).
          Hide
          thetaphi Uwe Schindler added a comment -

          I rewrote the WIndows startup script, but I stumbled on an issue with the Java 9 command line parser. I asked on the Hotspot Mailing List:
          http://mail.openjdk.java.net/pipermail/hotspot-dev/2017-August/027962.html

          I am currently adapting Apache Solr's startup scripts for Java 9. Linux was already done at the beginning of this year and works perfectly, but Windows brings some problems. I already fixed version number parsing, but I stumbled on the following: In the Windows ".cmd" shell script it uses the following to enable Garbage collection logging to a separate file, if Java 9 is detected:
          set GC_LOG_OPTS="-Xlog:gc*:file=Unable to render embedded object: File (SOLR_LOGS_DIR) not found.\solr_gc.log:time,uptime:filecount=9,filesize=20000"

          The problem is now that "Unable to render embedded object: File (SOLR_LOGS_DIR) not found." is already expanded to an absolute Windows Path by the shell and therefore starts with "C:\". The problem is now the colon, which breaks the log parsing. When Java 9 starts it exits with the following parsing error:
          Invalid -Xlog option '-Xlog:gc*:file=C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\logs\solr_gc.log:time,uptime:filecount=9,filesize=20000'

          If I replace with a simple file name, without path/drive letter it works. How to escape the colon in the drive letter correctly, to me this looks like a bummer?

          Show
          thetaphi Uwe Schindler added a comment - I rewrote the WIndows startup script, but I stumbled on an issue with the Java 9 command line parser. I asked on the Hotspot Mailing List: http://mail.openjdk.java.net/pipermail/hotspot-dev/2017-August/027962.html I am currently adapting Apache Solr's startup scripts for Java 9. Linux was already done at the beginning of this year and works perfectly, but Windows brings some problems. I already fixed version number parsing, but I stumbled on the following: In the Windows ".cmd" shell script it uses the following to enable Garbage collection logging to a separate file, if Java 9 is detected: set GC_LOG_OPTS="-Xlog:gc*:file= Unable to render embedded object: File (SOLR_LOGS_DIR) not found. \solr_gc.log:time,uptime:filecount=9,filesize=20000" The problem is now that " Unable to render embedded object: File (SOLR_LOGS_DIR) not found. " is already expanded to an absolute Windows Path by the shell and therefore starts with "C:\". The problem is now the colon, which breaks the log parsing. When Java 9 starts it exits with the following parsing error: Invalid -Xlog option '-Xlog:gc*:file=C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\logs\solr_gc.log:time,uptime:filecount=9,filesize=20000' If I replace with a simple file name, without path/drive letter it works. How to escape the colon in the drive letter correctly, to me this looks like a bummer?
          Hide
          thetaphi Uwe Schindler added a comment -

          Here my current patch, which breaks, because of the absolute path issue.

          Show
          thetaphi Uwe Schindler added a comment - Here my current patch, which breaks, because of the absolute path issue.
          Hide
          thetaphi Uwe Schindler added a comment -

          BTW, I set this as blocker, as we cannot release Solr 7 at the same time like Java 9 and it won't work.
          If we cannot solve the logging issue, I will comment out the separate log file on windows...

          Show
          thetaphi Uwe Schindler added a comment - BTW, I set this as blocker, as we cannot release Solr 7 at the same time like Java 9 and it won't work. If we cannot solve the logging issue, I will comment out the separate log file on windows...
          Hide
          thetaphi Uwe Schindler added a comment -

          Hi,
          this variant works, although it looks crazy: The \" escaping is ignored by the shell of windows, so it is passed as-is to Java (because "\" is not a good escape char on windows, so this approach won't work on Linux, but there are not colons in filenames!)

          This patch does not do the same stuff like Linux, because we do not have easy regular expressions available in Windows. The GC_LOG_OPTS used as it was before the change on Java <= 8, but for Java 9 it is ignored. I have no better idea how to do this. IMHO, this is better than not working at all! I added documentation for this.

          Does anybody have a better idea or knows how to do the same magic regex replcaments like in the linux shell script? Is the current approach acceptable, Hoss Man?

          Show
          thetaphi Uwe Schindler added a comment - Hi, this variant works, although it looks crazy: The \" escaping is ignored by the shell of windows, so it is passed as-is to Java (because "\" is not a good escape char on windows, so this approach won't work on Linux, but there are not colons in filenames!) This patch does not do the same stuff like Linux, because we do not have easy regular expressions available in Windows. The GC_LOG_OPTS used as it was before the change on Java <= 8, but for Java 9 it is ignored. I have no better idea how to do this. IMHO, this is better than not working at all! I added documentation for this. Does anybody have a better idea or knows how to do the same magic regex replcaments like in the linux shell script? Is the current approach acceptable, Hoss Man ?
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          I reordered the checks. Java 9 is detected before IBM J9. Because IBM J9 for Java 9 will handle the command line options like Oracle. This goes in-line with Hoss Man's UNIX shell script.

          I just noticed: JAVA_BUILD is not used anywhere, may I remove it? Because this wouldn't ever work with Java 9, as build numbers are handled by other means (no "_"). The UNIX shell script does not have any build version parsing, so I think it's a relict from former times.

          Show
          thetaphi Uwe Schindler added a comment - - edited I reordered the checks. Java 9 is detected before IBM J9. Because IBM J9 for Java 9 will handle the command line options like Oracle. This goes in-line with Hoss Man 's UNIX shell script. I just noticed: JAVA_BUILD is not used anywhere, may I remove it? Because this wouldn't ever work with Java 9, as build numbers are handled by other means (no "_"). The UNIX shell script does not have any build version parsing, so I think it's a relict from former times.
          Hide
          hossman Hoss Man added a comment -

          My thoughts in no particular order – w/o having looked at /understood the patch (i'm not familar with most windows batch file syntax)...

          BTW, I set this as blocker, as we cannot release Solr 7 at the same time like Java 9 and it won't work.

          I don't think "solr + windows + jdk9" is an important enough use case to warrant holding up lucene/solr 7. We can release note that solr.cmd startup script doesn't work in that permutation and focus on fixing later (if anyone is really hardcore about using solr + windows + jdk9 they can probably work around using the solr.cmd script)

          The GC_LOG_OPTS used as it was before the change on Java <= 8, but for Java 9 it is ignored. I have no better idea how to do this. IMHO, this is better than not working at all! I added documentation for this.

          My personal opinion is that it would be better to continue to support things as best we can for existing solr users who may upgrade solr w/o caring about jdk9 at all – ie: keep supporting/using GC_LOG_OPTS – but in the special in the case where they use jdk9 and try to specify a value for the GC_LOG_OPTS ... we should hard fail w/message that solr can't support GC_LOG_OPTS when using jdk9.

          I believe you said in your current patch you "ignored" GC_LOG_OPTS when using jdk9? ... i think that's a mistake, as existing solr users who may have already set GC_LOG_OPTS and then later upgrade to jdk9 will get confusing silently different behavior. better to fail fast and be clear that they can't do that (unless/until we find a better solution)

          Show
          hossman Hoss Man added a comment - My thoughts in no particular order – w/o having looked at /understood the patch (i'm not familar with most windows batch file syntax)... BTW, I set this as blocker, as we cannot release Solr 7 at the same time like Java 9 and it won't work. I don't think "solr + windows + jdk9" is an important enough use case to warrant holding up lucene/solr 7. We can release note that solr.cmd startup script doesn't work in that permutation and focus on fixing later (if anyone is really hardcore about using solr + windows + jdk9 they can probably work around using the solr.cmd script) The GC_LOG_OPTS used as it was before the change on Java <= 8, but for Java 9 it is ignored. I have no better idea how to do this. IMHO, this is better than not working at all! I added documentation for this. My personal opinion is that it would be better to continue to support things as best we can for existing solr users who may upgrade solr w/o caring about jdk9 at all – ie: keep supporting/using GC_LOG_OPTS – but in the special in the case where they use jdk9 and try to specify a value for the GC_LOG_OPTS ... we should hard fail w/message that solr can't support GC_LOG_OPTS when using jdk9. I believe you said in your current patch you "ignored" GC_LOG_OPTS when using jdk9? ... i think that's a mistake, as existing solr users who may have already set GC_LOG_OPTS and then later upgrade to jdk9 will get confusing silently different behavior. better to fail fast and be clear that they can't do that (unless/until we find a better solution)
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          Patch with Hoss Man's suggestions. If somebody sets GC_LOG_OPTS on Java 9 it bails out. I also updated documentation in the solr.in.cmd file.

          I tested with:

          • Java 8u144: Works as usual, GC_LOG_OPTS is respected if explicitely set
          • Java 9b178: Works now by default; if you comment out the GC_LOG_OPTS it fails early. BTW, in the UNIX scripts it would also fail if somebody updates Java 8 to Java 9, because the logging options are incompatible. So we are consistent for users that have GC_LOG_OPTS configured and migrate.
          • IBM J9 (Java 8): Works as before
          • Java 7u90: Fails early
          • Java 6: Fails early

          I think we should really commit this and maybe later improve this. It looks like I am the only active Solr committer that knows windows shell scripts a bit. IMHO, maybe we should switch to PowerShell, really! PowerShell is now installed on all supported Windows VMs, Windows 7 is out of service.

          Hoss Man: Any complaints or do you trust me? The current state is much better than before and the added code is trivial. IMHO, we should really not release that without basic Java 9 support and some migration path. Java comes out on Sept 21 (for sure, I already booked my tickets to the party in Munich).

          Show
          thetaphi Uwe Schindler added a comment - - edited Patch with Hoss Man 's suggestions. If somebody sets GC_LOG_OPTS on Java 9 it bails out. I also updated documentation in the solr.in.cmd file. I tested with: Java 8u144: Works as usual, GC_LOG_OPTS is respected if explicitely set Java 9b178: Works now by default; if you comment out the GC_LOG_OPTS it fails early. BTW, in the UNIX scripts it would also fail if somebody updates Java 8 to Java 9, because the logging options are incompatible. So we are consistent for users that have GC_LOG_OPTS configured and migrate. IBM J9 (Java 8): Works as before Java 7u90: Fails early Java 6: Fails early I think we should really commit this and maybe later improve this. It looks like I am the only active Solr committer that knows windows shell scripts a bit. IMHO, maybe we should switch to PowerShell, really! PowerShell is now installed on all supported Windows VMs, Windows 7 is out of service. Hoss Man : Any complaints or do you trust me? The current state is much better than before and the added code is trivial. IMHO, we should really not release that without basic Java 9 support and some migration path. Java comes out on Sept 21 (for sure, I already booked my tickets to the party in Munich).
          Hide
          thetaphi Uwe Schindler added a comment -

          And BTW, it also works with whitespace in Solr's installation directory.

          Show
          thetaphi Uwe Schindler added a comment - And BTW, it also works with whitespace in Solr's installation directory.
          Hide
          thetaphi Uwe Schindler added a comment -

          FYI, here is the Java 9 output:

          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr start -e techproducts -verbose
          Using Solr root directory: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr
          Using Java: C:\Program Files\Java\jdk-9\bin\java
          java version "9"
          Java(TM) SE Runtime Environment (build 9+178)
          Java HotSpot(TM) 64-Bit Server VM (build 9+178, mixed mode)
          
          Running with
          serverDir=C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server,
          exampleDir=C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example
          script=C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin\solr.cmd
          Creating Solr home directory C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr
          
          Starting up Solr on port 8983 using command:
          "C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin\solr.cmd" start -p 8983 -s "C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr"
          
          Using Solr root directory: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr
          Using Java: C:\Program Files\Java\jdk-9\bin\java
          java version "9"
          Java(TM) SE Runtime Environment (build 9+178)
          Java HotSpot(TM) 64-Bit Server VM (build 9+178, mixed mode)
          
          Starting Solr using the following settings:
              JAVA            = C:\Program Files\Java\jdk-9\bin\java
              SOLR_SERVER_DIR = C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server
              SOLR_HOME       = C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr
              SOLR_HOST       =
              SOLR_PORT       = 8983
              STOP_PORT       = 7983
              SOLR_JAVA_MEM   = -Xms512m -Xmx512m
              GC_TUNE         = -XX:NewRatio=3    -XX:SurvivorRatio=4    -XX:TargetSurvivorRatio=90    -XX:MaxTenuringThreshold=8    -XX:+UseConcMarkSweepGC    -XX:+UseParNewGC    -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4    -XX:+CMSScavengeBeforeRemark    -XX:PretenureSizeThreshold=64m    -XX:+UseCMSInitiatingOccupancyOnly    -XX:CMSInitiatingOccupancyFraction=50    -XX:CMSMaxAbortablePrecleanTime=6000    -XX:+CMSParallelRemarkEnabled    -XX:+ParallelRefProcEnabled    -XX:-OmitStackTraceInFastThrow
              GC_LOG_OPTS     = "-Xlog:gc*:file=\"C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr\..\logs\solr_gc.log\":time,uptime:filecount=9,filesize=20000"
              SOLR_TIMEZONE   = UTC
              SOLR_OPTS       = -Xss256k
          
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was deprecated in version 9.0 and will likely be removed in a future release.
          
          Checking status of Solr at http://localhost:8983/solr ...
          Waiting up to 30 to see Solr running on port 8983
          Started Solr server on port 8983. Happy searching!
          
          Solr is running on 8983 in standalone mode with status:
          {
            "solr_home":"C:\\Users\\Uwe Schindler\\Projects\\lucene\\trunk-lusolr1\\solr\\example\\techproducts\\solr",
            "version":"8.0.0-SNAPSHOT a7f2c30b405cb5d131826cf6c9ea5fe8a7cc3d71 - Uwe Schindler - 2017-08-20 19:20:02",
            "startTime":"2017-08-21T19:21:56.240Z",
            "uptime":"0 days, 0 hours, 0 minutes, 3 seconds",
            "memory":"58.9 MB (%12) of 490.7 MB",
            "baseUrl":"http://localhost:8983/solr"}
          
          Copying configuration to new core instance directory:
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr\techproducts
          
          Creating new core 'techproducts' using command:
          http://localhost:8983/solr/admin/cores?action=CREATE&name=techproducts&instanceDir=techproducts
          
          {
            "responseHeader":{
              "status":0,
              "QTime":2529},
            "core":"techproducts"}
          
          
          Indexing tech product example docs from C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\exampledocs
          SimplePostTool version 5.0.0
          Posting files to [base] url http://localhost:8983/solr/techproducts/update using content-type application/xml...
          POSTing file gb18030-example.xml to [base]
          POSTing file hd.xml to [base]
          POSTing file ipod_other.xml to [base]
          POSTing file ipod_video.xml to [base]
          POSTing file manufacturers.xml to [base]
          POSTing file mem.xml to [base]
          POSTing file money.xml to [base]
          POSTing file monitor.xml to [base]
          POSTing file monitor2.xml to [base]
          POSTing file mp500.xml to [base]
          POSTing file sd500.xml to [base]
          POSTing file solr.xml to [base]
          POSTing file utf8-example.xml to [base]
          POSTing file vidcard.xml to [base]
          14 files indexed.
          COMMITting Solr index changes to http://localhost:8983/solr/techproducts/update...
          Time spent: 0:00:00.689
          
          Solr techproducts example launched successfully. Direct your Web browser to http://localhost:8983/solr to visit the Solr Admin UI
          
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr stop -all
          Stopping Solr process 17596 running on port 8983
          
          Gewartet wird 0 Sekunden. Weiter mit beliebiger Taste...
          
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>
          
          Show
          thetaphi Uwe Schindler added a comment - FYI, here is the Java 9 output: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr start -e techproducts -verbose Using Solr root directory: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr Using Java: C:\Program Files\Java\jdk-9\bin\java java version "9" Java(TM) SE Runtime Environment (build 9+178) Java HotSpot(TM) 64-Bit Server VM (build 9+178, mixed mode) Running with serverDir=C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server, exampleDir=C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example script=C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin\solr.cmd Creating Solr home directory C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr Starting up Solr on port 8983 using command: "C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin\solr.cmd" start -p 8983 -s "C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr" Using Solr root directory: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr Using Java: C:\Program Files\Java\jdk-9\bin\java java version "9" Java(TM) SE Runtime Environment (build 9+178) Java HotSpot(TM) 64-Bit Server VM (build 9+178, mixed mode) Starting Solr using the following settings: JAVA = C:\Program Files\Java\jdk-9\bin\java SOLR_SERVER_DIR = C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server SOLR_HOME = C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr SOLR_HOST = SOLR_PORT = 8983 STOP_PORT = 7983 SOLR_JAVA_MEM = -Xms512m -Xmx512m GC_TUNE = -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:-OmitStackTraceInFastThrow GC_LOG_OPTS = "-Xlog:gc*:file=\"C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr\..\logs\solr_gc.log\":time,uptime:filecount=9,filesize=20000" SOLR_TIMEZONE = UTC SOLR_OPTS = -Xss256k Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was deprecated in version 9.0 and will likely be removed in a future release. Checking status of Solr at http://localhost:8983/solr ... Waiting up to 30 to see Solr running on port 8983 Started Solr server on port 8983. Happy searching! Solr is running on 8983 in standalone mode with status: { "solr_home":"C:\\Users\\Uwe Schindler\\Projects\\lucene\\trunk-lusolr1\\solr\\example\\techproducts\\solr", "version":"8.0.0-SNAPSHOT a7f2c30b405cb5d131826cf6c9ea5fe8a7cc3d71 - Uwe Schindler - 2017-08-20 19:20:02", "startTime":"2017-08-21T19:21:56.240Z", "uptime":"0 days, 0 hours, 0 minutes, 3 seconds", "memory":"58.9 MB (%12) of 490.7 MB", "baseUrl":"http://localhost:8983/solr"} Copying configuration to new core instance directory: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\techproducts\solr\techproducts Creating new core 'techproducts' using command: http://localhost:8983/solr/admin/cores?action=CREATE&name=techproducts&instanceDir=techproducts { "responseHeader":{ "status":0, "QTime":2529}, "core":"techproducts"} Indexing tech product example docs from C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\example\exampledocs SimplePostTool version 5.0.0 Posting files to [base] url http://localhost:8983/solr/techproducts/update using content-type application/xml... POSTing file gb18030-example.xml to [base] POSTing file hd.xml to [base] POSTing file ipod_other.xml to [base] POSTing file ipod_video.xml to [base] POSTing file manufacturers.xml to [base] POSTing file mem.xml to [base] POSTing file money.xml to [base] POSTing file monitor.xml to [base] POSTing file monitor2.xml to [base] POSTing file mp500.xml to [base] POSTing file sd500.xml to [base] POSTing file solr.xml to [base] POSTing file utf8-example.xml to [base] POSTing file vidcard.xml to [base] 14 files indexed. COMMITting Solr index changes to http://localhost:8983/solr/techproducts/update... Time spent: 0:00:00.689 Solr techproducts example launched successfully. Direct your Web browser to http://localhost:8983/solr to visit the Solr Admin UI C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr stop -all Stopping Solr process 17596 running on port 8983 Gewartet wird 0 Sekunden. Weiter mit beliebiger Taste... C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>
          Hide
          thetaphi Uwe Schindler added a comment -

          I will test with latest Java 9 in a minute, but the recent builds did not change anymore.

          Show
          thetaphi Uwe Schindler added a comment - I will test with latest Java 9 in a minute, but the recent builds did not change anymore.
          Hide
          hossman Hoss Man added a comment -

          Hoss Man: Any complaints or do you trust me?

          I trust you ... if you think it's ready, i believe you – heavy commit and let's move on.
          If you don't think it's ready then i say it shouldn't hold up the 7.0 release.

          Show
          hossman Hoss Man added a comment - Hoss Man: Any complaints or do you trust me? I trust you ... if you think it's ready, i believe you – heavy commit and let's move on. If you don't think it's ready then i say it shouldn't hold up the 7.0 release.
          Hide
          thetaphi Uwe Schindler added a comment -

          Latest build (build 181) of Java 9 - the official RC - also works:

          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr start -verbose
          Using Solr root directory: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr
          Using Java: C:\Program Files\Java\jdk-9\bin\java
          java version "9"
          Java(TM) SE Runtime Environment (build 9+181)
          Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)
          
          Archiving 1 old GC log files to C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\logs\archived
          Archiving 1 console log files to C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\logs\archived
          Rotating solr logs, keeping a max of 9 generations
          Starting Solr using the following settings:
              JAVA            = C:\Program Files\Java\jdk-9\bin\java
              SOLR_SERVER_DIR = C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server
              SOLR_HOME       = C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\solr
              SOLR_HOST       =
              SOLR_PORT       = 8983
              STOP_PORT       = 7983
              SOLR_JAVA_MEM   = -Xms512m -Xmx512m
              GC_TUNE         = -XX:NewRatio=3    -XX:SurvivorRatio=4    -XX:TargetSurvivorRatio=90    -XX:MaxTenuringThreshold=8    -XX:+UseConcMarkSweepGC    -XX:+UseParNewGC    -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4    -XX:+CMSScavengeBeforeRemark    -XX:PretenureSizeThreshold=64m    -XX:+UseCMSInitiatingOccupancyOnly    -XX:CMSInitiatingOccupancyFraction=50    -XX:CMSMaxAbortablePrecleanTime=6000    -XX:+CMSParallelRemarkEnabled    -XX:+ParallelRefProcEnabled    -XX:-OmitStackTraceInFastThrow
              GC_LOG_OPTS     = "-Xlog:gc*:file=\"C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\logs\solr_gc.log\":time,uptime:filecount=9,filesize=20000"
              SOLR_TIMEZONE   = UTC
              SOLR_OPTS       = -Xss256k
          
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was deprecated in version 9.0 and will likely be removed in a future release.
          Waiting up to 30 to see Solr running on port 8983
          Started Solr server on port 8983. Happy searching!
          
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr stop -all
          Stopping Solr process 9572 running on port 8983
          
          Gewartet wird 0 Sekunden. Weiter mit beliebiger Taste...
          
          C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>
          
          Show
          thetaphi Uwe Schindler added a comment - Latest build (build 181) of Java 9 - the official RC - also works: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr start -verbose Using Solr root directory: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr Using Java: C:\Program Files\Java\jdk-9\bin\java java version "9" Java(TM) SE Runtime Environment (build 9+181) Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode) Archiving 1 old GC log files to C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\logs\archived Archiving 1 console log files to C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\logs\archived Rotating solr logs, keeping a max of 9 generations Starting Solr using the following settings: JAVA = C:\Program Files\Java\jdk-9\bin\java SOLR_SERVER_DIR = C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server SOLR_HOME = C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\solr SOLR_HOST = SOLR_PORT = 8983 STOP_PORT = 7983 SOLR_JAVA_MEM = -Xms512m -Xmx512m GC_TUNE = -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:-OmitStackTraceInFastThrow GC_LOG_OPTS = "-Xlog:gc*:file=\"C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\server\logs\solr_gc.log\":time,uptime:filecount=9,filesize=20000" SOLR_TIMEZONE = UTC SOLR_OPTS = -Xss256k Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was deprecated in version 9.0 and will likely be removed in a future release. Waiting up to 30 to see Solr running on port 8983 Started Solr server on port 8983. Happy searching! C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>solr stop -all Stopping Solr process 9572 running on port 8983 Gewartet wird 0 Sekunden. Weiter mit beliebiger Taste... C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr1\solr\bin>
          Hide
          thetaphi Uwe Schindler added a comment -

          OK, I will commit this a bit later.

          Maybe somebody has time to test it in addition to myself.

          Show
          thetaphi Uwe Schindler added a comment - OK, I will commit this a bit later. Maybe somebody has time to test it in addition to myself.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 86f7d6779a8fee56e4497fde7d8936e916b00814 in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=86f7d67 ]

          SOLR-8689: Fix bin/solr.cmd so it can run properly on Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 86f7d6779a8fee56e4497fde7d8936e916b00814 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=86f7d67 ] SOLR-8689 : Fix bin/solr.cmd so it can run properly on Java 9
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 88d04b239f2321e026b774298febd0e478a8f155 in lucene-solr's branch refs/heads/branch_7x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=88d04b2 ]

          SOLR-8689: Fix bin/solr.cmd so it can run properly on Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 88d04b239f2321e026b774298febd0e478a8f155 in lucene-solr's branch refs/heads/branch_7x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=88d04b2 ] SOLR-8689 : Fix bin/solr.cmd so it can run properly on Java 9
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ef0f443d4cf002bc24576f1e72556fa06584365f in lucene-solr's branch refs/heads/branch_7_0 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ef0f443 ]

          SOLR-8689: Fix bin/solr.cmd so it can run properly on Java 9

          1. Conflicts:
          2. solr/CHANGES.txt
          Show
          jira-bot ASF subversion and git services added a comment - Commit ef0f443d4cf002bc24576f1e72556fa06584365f in lucene-solr's branch refs/heads/branch_7_0 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ef0f443 ] SOLR-8689 : Fix bin/solr.cmd so it can run properly on Java 9 Conflicts: solr/CHANGES.txt
          Hide
          thetaphi Uwe Schindler added a comment -

          I did some minor changes in the config.in files (updates in comments, because "Java 8 and lower" is no longer applicable, Solr is minimum Java 8).
          I committed to master, 7.x and 7.0.

          If somebody wants this also in 6.6.1, please reopen!

          Show
          thetaphi Uwe Schindler added a comment - I did some minor changes in the config.in files (updates in comments, because "Java 8 and lower" is no longer applicable, Solr is minimum Java 8). I committed to master, 7.x and 7.0. If somebody wants this also in 6.6.1, please reopen!
          Hide
          thetaphi Uwe Schindler added a comment -

          I just noticed, the CHANGES.txt is out of sync, before release, somebody should update the 7.0 section so its identical everywhere.

          Show
          thetaphi Uwe Schindler added a comment - I just noticed, the CHANGES.txt is out of sync, before release, somebody should update the 7.0 section so its identical everywhere.
          Hide
          varunthacker Varun Thacker added a comment -

          Re-opening to backport to branch_6_6

          Show
          varunthacker Varun Thacker added a comment - Re-opening to backport to branch_6_6
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d30ae83426d7cf12784090b85ba611697165508d in lucene-solr's branch refs/heads/branch_6_6 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d30ae83 ]

          SOLR-8689: Fix bin/solr.cmd so it can run properly on Java 9

          (cherry picked from commit 86f7d67)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d30ae83426d7cf12784090b85ba611697165508d in lucene-solr's branch refs/heads/branch_6_6 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d30ae83 ] SOLR-8689 : Fix bin/solr.cmd so it can run properly on Java 9 (cherry picked from commit 86f7d67)
          Hide
          varunthacker Varun Thacker added a comment -

          I used my colleagues windows machine to test the bin/solr.cmd script after the commit on branch_6_6 with Java 9

          it runs fine but I see a warning related to Jetty

          F:\SOFTWARES\SOLR\Sources\lucene-solr\solr\bin>solr start -c -m 1g
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was deprecated in version 9.0 and will likely be removed in a future release.
          Waiting up to 30 to see Solr running on port 8983
          WARNING: An illegal reflective access operation has occurred
          WARNING: Illegal reflective access by org.eclipse.jetty.util.BufferUtil (file:/F:/SOFTWARES/SOLR/Sources/lucene-solr/solr/server/lib/jetty-util-9.3.14.v20161028.jar) to field java.nio.MappedByteBuffer.fd
          WARNING: Please consider reporting this to the maintainers of org.eclipse.jetty.util.BufferUtil
          WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
          WARNING: All illegal access operations will be denied in a future release
          Started Solr server on port 8983. Happy searching!
          
          Show
          varunthacker Varun Thacker added a comment - I used my colleagues windows machine to test the bin/solr.cmd script after the commit on branch_6_6 with Java 9 it runs fine but I see a warning related to Jetty F:\SOFTWARES\SOLR\Sources\lucene-solr\solr\bin>solr start -c -m 1g Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was deprecated in version 9.0 and will likely be removed in a future release. Waiting up to 30 to see Solr running on port 8983 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.eclipse.jetty.util.BufferUtil (file:/F:/SOFTWARES/SOLR/Sources/lucene-solr/solr/server/lib/jetty-util-9.3.14.v20161028.jar) to field java.nio.MappedByteBuffer.fd WARNING: Please consider reporting this to the maintainers of org.eclipse.jetty.util.BufferUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Started Solr server on port 8983. Happy searching!
          Hide
          varunthacker Varun Thacker added a comment -

          Thanks Uwe for tackling this. The fix will also be in 6.6.1 now

          Show
          varunthacker Varun Thacker added a comment - Thanks Uwe for tackling this. The fix will also be in 6.6.1 now
          Hide
          thetaphi Uwe Schindler added a comment -

          Hi Varun,
          the warnings are:

          • The garbage collectors are deprecated. That's something we should investigate. Maybe it's time to switch to G1GC at some time by default.
          • The BufferUtils warning is caused by reflection of Jetty into JDK internals. On tests we don't see this, because we execute Java with --illegal-access=deny. By default, Java 9 allows illegal accesses. IMHO, we should maybe also deny illegal access in Solr by default through the startup scripts. Jetty can handle this, so it falls back to something that does not try to reflect into JDK internals.

          I think we should open a separate issue and hack the startup scripts of solr to pass "--illegal-access=deny" to the JVM. Interestingly, I don't see this warning on master and 7.x, but on 7.0 and 6.6, so maybe that's just an issue of older Jetty versions (9.3.14 vs. 9.3.20).

          Show
          thetaphi Uwe Schindler added a comment - Hi Varun, the warnings are: The garbage collectors are deprecated. That's something we should investigate. Maybe it's time to switch to G1GC at some time by default. The BufferUtils warning is caused by reflection of Jetty into JDK internals. On tests we don't see this, because we execute Java with --illegal-access=deny. By default, Java 9 allows illegal accesses. IMHO, we should maybe also deny illegal access in Solr by default through the startup scripts. Jetty can handle this, so it falls back to something that does not try to reflect into JDK internals. I think we should open a separate issue and hack the startup scripts of solr to pass "--illegal-access=deny" to the JVM. Interestingly, I don't see this warning on master and 7.x, but on 7.0 and 6.6, so maybe that's just an issue of older Jetty versions (9.3.14 vs. 9.3.20).
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          That warning is this issue: https://github.com/eclipse/jetty.project/issues/1262 (fixed in Jetty 9.3.16), so it's fixed with Solr 7.1.

          Show
          thetaphi Uwe Schindler added a comment - - edited That warning is this issue: https://github.com/eclipse/jetty.project/issues/1262 (fixed in Jetty 9.3.16), so it's fixed with Solr 7.1.
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Bulk close after 7.1.0 release

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Bulk close after 7.1.0 release

            People

            • Assignee:
              thetaphi Uwe Schindler
              Reporter:
              thetaphi Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development