Derby
  1. Derby
  2. DERBY-5051

Update the docs to include the additional information available in derby.log. This information will be useful from product supportability stand point

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.8.1.2
    • Fix Version/s: 10.8.1.2
    • Component/s: Documentation
    • Labels:
      None

      Description

      With DERBY-4853, we log additional information in derby.log to make it easier for the support group to work with customer problems.

      Following 2 additional properties will now always be put in derby.log (every time a database is booted)
      user.dir
      derby.system.home

      user.dir is the system property which gives the user working directory.

      If the user has not specified derby,system.home, then Derby will look for optional derby.properties file in user.dir . If user has provided derby.system.home, then the optional derby.properties will be looked in derby.system.home rather than user.dir

      If user has not provided derby.system.home, then a sample of the 2 properties on derby.log looks as follows on my machine
      user.dir=C:\p4clients\svnmain\client2\trunk\systest\out142
      derby.system.home=null

      If user has set derby.system.home to c:./temp, then a sample of the 2 properties on derby.log looks as follows on my machine
      user.dir=C:\p4clients\svnmain\client2\trunk\systest\out142
      derby.system.home=c:/temp

      In addition to the above 2 properties, there can be following 3 optional properties in the log (following the 2 properties mentioned above)
      derby.stream.error.file
      derby.stream.error.method
      derby.stream.error.field

      The above 3 properties provide a way to alter where error messages get logged rather than tranditional derby.log file. In the absence of the above 3 optional properties, the location of derby.log follows the same logic as derby.properties. But if the user has set any or all of the 3 optional properties, then derby.log will not be in same location as derby.properties. In the Derby code, derby.stream.error.file takes precedence over the other 2 properties and derby.stream.error.method take precedence over the last property.

      1. DERBY-5051-2.zip
        10 kB
        Kim Haase
      2. DERBY-5051-2.stat
        0.1 kB
        Kim Haase
      3. DERBY-5051-2.diff
        4 kB
        Kim Haase
      4. DERBY-5051.zip
        26 kB
        Kim Haase
      5. DERBY-5051.stat
        0.4 kB
        Kim Haase
      6. DERBY-5051.diff
        20 kB
        Kim Haase

        Issue Links

          Activity

          Hide
          Kim Haase added a comment -

          Committed patch DERBY-5051-2.diff to documentation trunk at revision 1080223.

          I like to run the Getting Started tutorial tasks manually so that the timestamps will look somewhat credible if readers actually look at them. I also run sysinfo manually for the Tools guide example, which takes next to no time.

          Setting one or more of the derby.stream.error properties is a bit too advanced for Getting Started or the introductory topics in the Developer's Guide. Since the output for that part is not likely to change between releases, a one-time example of one or more of them for the Reference Manual is all that's needed, for just a snippet of the log output instead of the whole thing.

          Show
          Kim Haase added a comment - Committed patch DERBY-5051 -2.diff to documentation trunk at revision 1080223. I like to run the Getting Started tutorial tasks manually so that the timestamps will look somewhat credible if readers actually look at them. I also run sysinfo manually for the Tools guide example, which takes next to no time. Setting one or more of the derby.stream.error properties is a bit too advanced for Getting Started or the introductory topics in the Developer's Guide. Since the output for that part is not likely to change between releases, a one-time example of one or more of them for the Reference Manual is all that's needed, for just a snippet of the log output instead of the whole thing.
          Hide
          Kim Haase added a comment -

          Committed patch DERBY-5051-2.diff to documentation trunk at revision 1080223.

          I like to run the Getting Started tutorial tasks manually so that the timestamps will look somewhat credible if readers actually look at them. I also run sysinfo manually for the Tools guide example, which takes next to no time.

          Setting one or more of the derby.stream.error properties is a bit too advanced for Getting Started or the introductory topics in the Developer's Guide. Since the output for that part is not likely to change between releases, a one-time example of one or more of them for the Reference Manual is all that's needed, for just a snippet of the log output instead of the whole thing.

          Show
          Kim Haase added a comment - Committed patch DERBY-5051 -2.diff to documentation trunk at revision 1080223. I like to run the Getting Started tutorial tasks manually so that the timestamps will look somewhat credible if readers actually look at them. I also run sysinfo manually for the Tools guide example, which takes next to no time. Setting one or more of the derby.stream.error properties is a bit too advanced for Getting Started or the introductory topics in the Developer's Guide. Since the output for that part is not likely to change between releases, a one-time example of one or more of them for the Reference Manual is all that's needed, for just a snippet of the log output instead of the whole thing.
          Hide
          Kathey Marsden added a comment -

          It would be nice to have a specific test case or tool that runs and generates the doc derby.log examples. I think we have something like that for the SQLStates that can change frequently. I think for that one it is part of the release process, which I know we don't want to grow, but just reducing the log update task to a quick cut and paste would be an improvement.

          Show
          Kathey Marsden added a comment - It would be nice to have a specific test case or tool that runs and generates the doc derby.log examples. I think we have something like that for the SQLStates that can change frequently. I think for that one it is part of the release process, which I know we don't want to grow, but just reducing the log update task to a quick cut and paste would be an improvement.
          Hide
          Kim Haase added a comment -

          Thanks, Mamta.

          We have a couple of simple examples of the derby log but none showing a redirect. The problem is that I have to update them every release, so the more examples we add, the more work this task requires. We show the new output that is there by default now. If you have an example that shows the relevant output with one of those property settings, please add it to this issue and maybe I could add it to the reference manual topic for that property. If we used just a snippet showing the user.dir and derby.system.home settings along with the derby.stream.... property, it would not need to change with each release.

          In the meantime I'll commit this patch.

          Show
          Kim Haase added a comment - Thanks, Mamta. We have a couple of simple examples of the derby log but none showing a redirect. The problem is that I have to update them every release, so the more examples we add, the more work this task requires. We show the new output that is there by default now. If you have an example that shows the relevant output with one of those property settings, please add it to this issue and maybe I could add it to the reference manual topic for that property. If we used just a snippet showing the user.dir and derby.system.home settings along with the derby.stream.... property, it would not need to change with each release. In the meantime I'll commit this patch.
          Hide
          Mamta A. Satoor added a comment -

          The latest set of changes look good, Kim.

          I am not sure if an actual example showing one or all of these properties in derby.log will be helpful or not. I think there are couple places in the doc, where we show how a sample derby.log looks like.

          Show
          Mamta A. Satoor added a comment - The latest set of changes look good, Kim. I am not sure if an actual example showing one or all of these properties in derby.log will be helpful or not. I think there are couple places in the doc, where we show how a sample derby.log looks like.
          Hide
          Kim Haase added a comment -

          Attaching DERBY-5051-2.diff, DERBY-5051-2.stat, and DERBY-5051-2.zip, with changes to the following files:

          M src/devguide/cdevdvlp25889.dita
          M src/ref/rrefproper35028.dita
          M src/ref/rrefproper33027.dita
          M src/ref/rrefproper18151.dita

          I added more info to the error log topic in the devguide and also did a bit of reorganizing into paragraphs and the shortdesc element, to make the information more visible.

          I also added log information to each ref manual topic.

          Please let me know if more changes would be helpful.

          Show
          Kim Haase added a comment - Attaching DERBY-5051 -2.diff, DERBY-5051 -2.stat, and DERBY-5051 -2.zip, with changes to the following files: M src/devguide/cdevdvlp25889.dita M src/ref/rrefproper35028.dita M src/ref/rrefproper33027.dita M src/ref/rrefproper18151.dita I added more info to the error log topic in the devguide and also did a bit of reorganizing into paragraphs and the shortdesc element, to make the information more visible. I also added log information to each ref manual topic. Please let me know if more changes would be helpful.
          Hide
          Kim Haase added a comment -

          Reopening to provide more complete information about log entries.

          Show
          Kim Haase added a comment - Reopening to provide more complete information about log entries.
          Hide
          Kim Haase added a comment -

          Oh, I see. the setting(s) of the property or properties shows up in the redirected log? I guess I didn't quite grasp that. Wouldn't you have to know where the log info was going already in order to even find the log with these settings in it? I suppose it might be helpful if you needed to send the log contents somewhere for diagnostic purposes.

          Anyway, I'll reopen this issue and put a mention in each of the Reference Manual entries as well as in the error log topic in the developer's guide.

          Show
          Kim Haase added a comment - Oh, I see. the setting(s) of the property or properties shows up in the redirected log? I guess I didn't quite grasp that. Wouldn't you have to know where the log info was going already in order to even find the log with these settings in it? I suppose it might be helpful if you needed to send the log contents somewhere for diagnostic purposes. Anyway, I'll reopen this issue and put a mention in each of the Reference Manual entries as well as in the error log topic in the developer's guide.
          Hide
          Mamta A. Satoor added a comment -

          I was thinking that we should mention somewhere that if any(all) of following 3 properties is set, it will be logged in derby.log. This info might be useful for support group when working with a problem
          derby.stream.error.file
          derby.stream.error.method
          derby.stream.error.field

          Show
          Mamta A. Satoor added a comment - I was thinking that we should mention somewhere that if any(all) of following 3 properties is set, it will be logged in derby.log. This info might be useful for support group when working with a problem derby.stream.error.file derby.stream.error.method derby.stream.error.field
          Hide
          Kim Haase added a comment -

          I mentioned all three of those properties in the topic "The error log" (cdevdvlp25889.dita), which previously mentioned only derby.stream.error.method. If you think it would be helpful to say more, or if any of the new language could use some fixes, please let me know.

          Show
          Kim Haase added a comment - I mentioned all three of those properties in the topic "The error log" (cdevdvlp25889.dita), which previously mentioned only derby.stream.error.method. If you think it would be helpful to say more, or if any of the new language could use some fixes, please let me know.
          Hide
          Mamta A. Satoor added a comment -

          Kim, thanks so much for working on this. The changes look great. I just had one question. I didn't see any mention of the following optional properties in derby.log
          derby.stream.error.file
          derby.stream.error.method
          derby.stream.error.field

          Maybe you documented it and I missed it. Thanks

          Show
          Mamta A. Satoor added a comment - Kim, thanks so much for working on this. The changes look great. I just had one question. I didn't see any mention of the following optional properties in derby.log derby.stream.error.file derby.stream.error.method derby.stream.error.field Maybe you documented it and I missed it. Thanks
          Hide
          Kim Haase added a comment -

          Thanks, Rick. I've committed this patch so it can get into the 10.8 RC. If any changes are needed, I can still make them on the trunk and if a second RC is needed, they can get into the release.

          Committed patch DERBY-5051.diff to documentation trunk at revision 1078171.

          Show
          Kim Haase added a comment - Thanks, Rick. I've committed this patch so it can get into the 10.8 RC. If any changes are needed, I can still make them on the trunk and if a second RC is needed, they can get into the release. Committed patch DERBY-5051 .diff to documentation trunk at revision 1078171.
          Hide
          Rick Hillegas added a comment -

          Hi Kim,

          I am not an expert on Mamta's changes but I have taken a quick look at your patch and it seems good to me. In particular, the extra blurb about the deregister property is welcome. Thanks.

          Show
          Rick Hillegas added a comment - Hi Kim, I am not an expert on Mamta's changes but I have taken a quick look at your patch and it seems good to me. In particular, the extra blurb about the deregister property is welcome. Thanks.
          Hide
          Kim Haase added a comment -

          Attaching DERBY-5051.diff, DERBY-5051.stat, and DERBY-5051.zip, containing edits to the following files. Not all the edits are strictly log-related – I hope that is all right? Explanations below.

          M src/tools/rtoolssysinfo1002629.dita
          M src/devguide/cdevdvlp27715.dita
          M src/devguide/tdevdvlp20349.dita
          M src/devguide/cdevdvlp25889.dita
          M src/getstart/tgssetupverify.dita
          M src/getstart/twwdactivity1.dita
          M src/getstart/twwdactivity3_Setup.dita
          M src/getstart/twwdactivity2.dita
          M src/getstart/twwdactivity4.dita

          I updated the contents of the derby.log output in cdevdvlp27715.dita (Booting databases), tdevdvlp20349.dita (Shutting down the system), and twwdactivity1.dita (Creating a Derby database and running SQL statements). I did not include the istat messages at the end of the shutdown log entry, since those are disappearing, for the time being at least.

          I added information about the log to cdevdvlp27715.dita and updated the information in cdevdvlp25889.dita (The error log). I hope this is correct.

          I hope it is okay that I added a few more things:

          I took the opportunity to add information about the deregister attribute to the shutdown topic (tdevdvlp20349.dita), since it seemed to belong there, and I had missed this topic in the patch for DERBY-5043.

          I also took the opportunity to make some updates that are not strictly log-related but that have to be made for every release – fixing the output for examples in the Getting Started and Tools guides. These do not use the final release number and build because that isn't available yet.

          Show
          Kim Haase added a comment - Attaching DERBY-5051 .diff, DERBY-5051 .stat, and DERBY-5051 .zip, containing edits to the following files. Not all the edits are strictly log-related – I hope that is all right? Explanations below. M src/tools/rtoolssysinfo1002629.dita M src/devguide/cdevdvlp27715.dita M src/devguide/tdevdvlp20349.dita M src/devguide/cdevdvlp25889.dita M src/getstart/tgssetupverify.dita M src/getstart/twwdactivity1.dita M src/getstart/twwdactivity3_Setup.dita M src/getstart/twwdactivity2.dita M src/getstart/twwdactivity4.dita I updated the contents of the derby.log output in cdevdvlp27715.dita (Booting databases), tdevdvlp20349.dita (Shutting down the system), and twwdactivity1.dita (Creating a Derby database and running SQL statements). I did not include the istat messages at the end of the shutdown log entry, since those are disappearing, for the time being at least. I added information about the log to cdevdvlp27715.dita and updated the information in cdevdvlp25889.dita (The error log). I hope this is correct. I hope it is okay that I added a few more things: I took the opportunity to add information about the deregister attribute to the shutdown topic (tdevdvlp20349.dita), since it seemed to belong there, and I had missed this topic in the patch for DERBY-5043 . I also took the opportunity to make some updates that are not strictly log-related but that have to be made for every release – fixing the output for examples in the Getting Started and Tools guides. These do not use the final release number and build because that isn't available yet.
          Hide
          Knut Anders Hatlen added a comment -

          I added a release note to DERBY-4939 that mentions the properties derby.storage.indexStats.auto and derby.storage.indexStats.log, so even if we don't get around to documenting them, at least there's some info available about them to those who download the release.

          But given that the logging is primarily meant for debugging, and the messages may confuse users and possibly make them believe there is a problem, I'm leaning towards disabling the istat logging by default. If users have problems that we suspect may have something to do with istat, we can always ask them to enable extra logging manually.

          Show
          Knut Anders Hatlen added a comment - I added a release note to DERBY-4939 that mentions the properties derby.storage.indexStats.auto and derby.storage.indexStats.log, so even if we don't get around to documenting them, at least there's some info available about them to those who download the release. But given that the logging is primarily meant for debugging, and the messages may confuse users and possibly make them believe there is a problem, I'm leaning towards disabling the istat logging by default. If users have problems that we suspect may have something to do with istat, we can always ask them to enable extra logging manually.
          Hide
          Kim Haase added a comment -

          Thanks for the explanation, Kristian. But uh-oh, I don't think we document the new index statistics feature – not yet anyway. I don't see a doc issue for the derby.storage.indexStats.log property. Should there be one? Is documentation needed elsewhere too?

          Show
          Kim Haase added a comment - Thanks for the explanation, Kristian. But uh-oh, I don't think we document the new index statistics feature – not yet anyway. I don't see a doc issue for the derby.storage.indexStats.log property. Should there be one? Is documentation needed elsewhere too?
          Hide
          Kristian Waagan added a comment -

          Hi Kim,

          That last line is the new index statistics update daemon printing its statistics on shutdown. It is cryptic and noone - myself being an exception I hope - will understand it...
          I wrote this code, but since I'm on leave right now the logging code hasn't been worked on as much as it should have. I think the line is important for early stage investigation/observation in the wild, but agree that is may cause confusion.

          Some improvements:
          a) Print it only if the daemon has actually done something.
          b) Document why the line is printed, i.e. for debugging/observability, and how it can be disabled (see below). Do we already have something written down for the new feature?
          c) Document the meaning of the various metrics.

          To give people a chance to judge the usefulness of this line, I'll briefly explain the metrics printed:

          • active : whether the worker thread was running when the daemon was shut down (in code: runningThread != null)
          • work : ms spent doing work
          • age : ms since daemon creation
          • q : work units currently queued
          • p : work units processed
          • s : work units scheduled
          • k : number of known/excpected errors encountered (i.e table is dropped while being scanned)
          • u : number of unexpected errors encountered (these will affect the current work unit, but the daemon will remain active)
          • c : number of consecutive errors encountered (daemon will be disabled if a threshold is exceeded)
          • f : rejections due to a full queue
          • d : rejections due to work unit duplicates
          • o : rejections due to other causes, which is currently only attempts to schedule a work unit after the daemon has been disabled

          Currently logging for the istat feature will be enabled by default, unless it has been specifically disabled by setting derby.storage.indexStats.log=false

          Show
          Kristian Waagan added a comment - Hi Kim, That last line is the new index statistics update daemon printing its statistics on shutdown. It is cryptic and noone - myself being an exception I hope - will understand it... I wrote this code, but since I'm on leave right now the logging code hasn't been worked on as much as it should have. I think the line is important for early stage investigation/observation in the wild, but agree that is may cause confusion. Some improvements: a) Print it only if the daemon has actually done something. b) Document why the line is printed, i.e. for debugging/observability, and how it can be disabled (see below). Do we already have something written down for the new feature? c) Document the meaning of the various metrics. To give people a chance to judge the usefulness of this line, I'll briefly explain the metrics printed: active : whether the worker thread was running when the daemon was shut down (in code: runningThread != null) work : ms spent doing work age : ms since daemon creation q : work units currently queued p : work units processed s : work units scheduled k : number of known/excpected errors encountered (i.e table is dropped while being scanned) u : number of unexpected errors encountered (these will affect the current work unit, but the daemon will remain active) c : number of consecutive errors encountered (daemon will be disabled if a threshold is exceeded) f : rejections due to a full queue d : rejections due to work unit duplicates o : rejections due to other causes, which is currently only attempts to schedule a work unit after the daemon has been disabled Currently logging for the istat feature will be enabled by default, unless it has been specifically disabled by setting derby.storage.indexStats.log=false
          Hide
          Kim Haase added a comment -

          I'll get to work on this, but I have a question. I looked at the new output after building derby from the code trunk.

          At least at present it seems that in addition to reporting the user.dir and derby.system.home settings, the log also contains the following at the end, after the "Shutting down ..." message:

          Mon Feb 28 14:47:23 EST 2011 Thread[main,5,main]

          {istat}

          stopping daemon, active=false, work/age=0/47446 [q/p/s=0/0/0,err:k/u/c=0/0/0,rej:f/d/o=0/0/0]

          What's this? Will it appear in the output of the released version?

          Show
          Kim Haase added a comment - I'll get to work on this, but I have a question. I looked at the new output after building derby from the code trunk. At least at present it seems that in addition to reporting the user.dir and derby.system.home settings, the log also contains the following at the end, after the "Shutting down ..." message: Mon Feb 28 14:47:23 EST 2011 Thread [main,5,main] {istat} stopping daemon, active=false, work/age=0/47446 [q/p/s=0/0/0,err:k/u/c=0/0/0,rej:f/d/o=0/0/0] What's this? Will it appear in the output of the released version?

            People

            • Assignee:
              Kim Haase
              Reporter:
              Mamta A. Satoor
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development