Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7506

Roll over GC logs by default via bin/solr scripts

    Details

      Description

      The Oracle JDK supports rolling over GC logs. I propose to add the following to the solr.in.

      {sh,cmd}

      scripts to enable it by default:

      -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M
      

      Unfortunately, the JDK doesn't have any option to append to existing log instead of overwriting so the latest log is overwritten. Maybe we can have the bin/solr script roll that after the process is killed?

      1. SOLR-7506.patch
        5 kB
        Jan Høydahl
      2. SOLR-7506.patch
        3 kB
        Jan Høydahl

        Issue Links

          Activity

          Hide
          andyetitmoves Ramkumar Aiyengar added a comment -

          Unfortunately, the JDK doesn't have any option to append to existing log instead of overwriting so the latest log is overwritten. Maybe we can have the bin/solr script roll that after the process is killed?

          We do backup the last gc log before we start? We probably can change that to roll to the same file name as what the JVM would have done.

          That apart, this should probably keep this similar to log4j defaults, i.e. 4 MB, and 9 logs.

          This probably also made me think about the way we are probably inconsistent with the number of logs and the log retention policy with the main Solr log. We currently ask log4j to retain 9 logs, but roll over to a timestamped file as well at startup – that will never be part of the retention policy..

          Show
          andyetitmoves Ramkumar Aiyengar added a comment - Unfortunately, the JDK doesn't have any option to append to existing log instead of overwriting so the latest log is overwritten. Maybe we can have the bin/solr script roll that after the process is killed? We do backup the last gc log before we start? We probably can change that to roll to the same file name as what the JVM would have done. That apart, this should probably keep this similar to log4j defaults, i.e. 4 MB, and 9 logs. This probably also made me think about the way we are probably inconsistent with the number of logs and the log retention policy with the main Solr log. We currently ask log4j to retain 9 logs, but roll over to a timestamped file as well at startup – that will never be part of the retention policy..
          Hide
          janhoy Jan Høydahl added a comment -

          Unfortunately the JVM is not that smart. It always starts writing a fresh solr_gc.log.0.current log file which is rolled into .0 without suffix. So after a restart, history gets lost even if we try to be smart. I'm working on something similar in SOLR-9570

          Show
          janhoy Jan Høydahl added a comment - Unfortunately the JVM is not that smart. It always starts writing a fresh solr_gc.log.0.current log file which is rolled into .0 without suffix. So after a restart, history gets lost even if we try to be smart. I'm working on something similar in SOLR-9570
          Hide
          janhoy Jan Høydahl added a comment -

          Patch

          • Enables rotation of GC logs per suggestion of Shalin. Keeps 9 generations, 20M each
          • Names the gc log solr_gc_YYYYmmddHHMMss.log, which will be suffixed with .N[.current] by the JRE

          NOTE: This should be applied on top of SOLR-9570, else we're creating a new problem with ever-increasing logs

          Show
          janhoy Jan Høydahl added a comment - Patch Enables rotation of GC logs per suggestion of Shalin. Keeps 9 generations, 20M each Names the gc log solr_gc_YYYYmmddHHMMss.log , which will be suffixed with .N[.current] by the JRE NOTE: This should be applied on top of SOLR-9570 , else we're creating a new problem with ever-increasing logs
          Hide
          janhoy Jan Høydahl added a comment -

          We don't necessarily need the timestamp in the file name, since we'll anyway move logs away into archived/ on every start, and delete the previous generation every time. An alternative to moving gc files to archived/ (SOLR-9570) could however be to leave them in SOLR_LOGS_DIR with timestamp in filename, and then on every restart a new set of rotated gc logs will be created, and then we delete solr_gc_* files >7days old on startup.

          Show
          janhoy Jan Høydahl added a comment - We don't necessarily need the timestamp in the file name, since we'll anyway move logs away into archived/ on every start, and delete the previous generation every time. An alternative to moving gc files to archived/ ( SOLR-9570 ) could however be to leave them in SOLR_LOGS_DIR with timestamp in filename, and then on every restart a new set of rotated gc logs will be created, and then we delete solr_gc_* files >7days old on startup.
          Hide
          janhoy Jan Høydahl added a comment -

          New patch

          • Adds GC log rotation with file name solr_gc.log
          • Merge with master
          • Updated UtilsTool to remove/rotate solr_gc.* as well as solr_gc_*

          Re-tested on mac, not Windows so far

          Show
          janhoy Jan Høydahl added a comment - New patch Adds GC log rotation with file name solr_gc.log Merge with master Updated UtilsTool to remove/rotate solr_gc.* as well as solr_gc_* Re-tested on mac, not Windows so far
          Hide
          janhoy Jan Høydahl added a comment -

          Tested on windows 10. Planning to commit during weekend.

          Show
          janhoy Jan Høydahl added a comment - Tested on windows 10. Planning to commit during weekend.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ef5737466e4597c21c80b167f1db295c081578d4 in lucene-solr's branch refs/heads/master from Jan Høydahl
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ef57374 ]

          SOLR-7506: Roll over GC logs by default via bin/solr scripts

          Show
          jira-bot ASF subversion and git services added a comment - Commit ef5737466e4597c21c80b167f1db295c081578d4 in lucene-solr's branch refs/heads/master from Jan Høydahl [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ef57374 ] SOLR-7506 : Roll over GC logs by default via bin/solr scripts
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ed203978fcb953f8196317c68eae18342f95cc44 in lucene-solr's branch refs/heads/branch_6x from Jan Høydahl
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ed20397 ]

          SOLR-7506: Roll over GC logs by default via bin/solr scripts

          (cherry picked from commit ef57374)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ed203978fcb953f8196317c68eae18342f95cc44 in lucene-solr's branch refs/heads/branch_6x from Jan Høydahl [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ed20397 ] SOLR-7506 : Roll over GC logs by default via bin/solr scripts (cherry picked from commit ef57374)
          Hide
          janhoy Jan Høydahl added a comment -

          This is now in. An improvement could be to keep the logs in archive/ for a few days instead of deleting on every startup...

          Show
          janhoy Jan Høydahl added a comment - This is now in. An improvement could be to keep the logs in archive/ for a few days instead of deleting on every startup...
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Closing after 6.3.0 release.

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.

            People

            • Assignee:
              janhoy Jan Høydahl
              Reporter:
              shalinmangar Shalin Shekhar Mangar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development