Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 10.4.1.3, 10.5.1.1
    • Component/s: JMX
    • Labels:
      None

      Description

      This is a draft requirement specification for adding monitoring and management extensions to Apache Derby using JMX. The requirements document has been uploaded on JIRA as well as the Derby Wiki page at http://wiki.apache.org/db-derby/_Requirement_Specifications_for_Monitoring_%26_Management_Extensions_using_JMX

      Developers and Users are requested to please look at the document (feature list in particular) and add their own rating to features by adding a coloumn to the table.

      Comments are welcome.

      1. derby1387_simple_9_1.txt
        14 kB
        Daniel John Debrunner
      2. DERBY-1387-1.diff
        72 kB
        Ole Gunnar Borstad
      3. DERBY-1387-1.stat
        1 kB
        Ole Gunnar Borstad
      4. DERBY-1387-2.diff
        72 kB
        Ole Gunnar Borstad
      5. DERBY-1387-2.stat
        1 kB
        Ole Gunnar Borstad
      6. DERBY-1387-3.diff
        74 kB
        Ole Gunnar Borstad
      7. DERBY-1387-3.stat
        1 kB
        Ole Gunnar Borstad
      8. DERBY-1387-4.diff
        75 kB
        Bernt M. Johnsen
      9. DERBY-1387-4.stat
        1 kB
        Bernt M. Johnsen
      10. DERBY-1387-5.diff
        86 kB
        Bernt M. Johnsen
      11. DERBY-1387-5.stat
        2 kB
        Bernt M. Johnsen
      12. DERBY-1387-6.zip
        16 kB
        Bernt M. Johnsen
      13. DERBY-1387-7.zip
        16 kB
        Bernt M. Johnsen
      14. DERBY-1387-8.zip
        14 kB
        Bernt M. Johnsen
      15. DERBY-1387-9.diff
        98 kB
        John H. Embretsen
      16. DERBY-1387-9.stat
        2 kB
        John H. Embretsen
      17. derbyjmx.patch
        137 kB
        Sanket Sharma
      18. jmx.diff
        74 kB
        Ole Gunnar Borstad
      19. jmx.stat
        1 kB
        Ole Gunnar Borstad
      20. jmxFuncspec.html
        74 kB
        John H. Embretsen
      21. jmxFuncspec.html
        70 kB
        John H. Embretsen
      22. jmxFuncspec.html
        67 kB
        John H. Embretsen
      23. jmxFuncspec.html
        45 kB
        John H. Embretsen
      24. jmxFuncspec.html
        42 kB
        John H. Embretsen
      25. jmxFuncspec.html
        20 kB
        Ole Gunnar Borstad
      26. jmxPolicyFileChanges_v1.diff
        3 kB
        John H. Embretsen
      27. Requirements for JMX.html
        190 kB
        Sanket Sharma
      28. Requirements for JMX.zip
        15 kB
        Sanket Sharma
      29. Requirements for JMX Updated.html
        190 kB
        Sanket Sharma

        Issue Links

          Activity

          Hide
          John H. Embretsen added a comment -

          Closing this issue.

          Show
          John H. Embretsen added a comment - Closing this issue.
          Hide
          John H. Embretsen added a comment -

          JMX Management and Monitoring functionality has been added to trunk and the 10.4 code branch. Marking this issue as resolved/fixed.

          Show
          John H. Embretsen added a comment - JMX Management and Monitoring functionality has been added to trunk and the 10.4 code branch. Marking this issue as resolved/fixed.
          Hide
          John H. Embretsen added a comment -

          Attaching version 3.2 of the functional spec for this feature, in an effort to make this relatively up-to-date regarding the 10.4.1 release.

          Show
          John H. Embretsen added a comment - Attaching version 3.2 of the functional spec for this feature, in an effort to make this relatively up-to-date regarding the 10.4.1 release.
          Hide
          Daniel John Debrunner added a comment -

          FYI on why the JDK 1.5 specific JMX permissions work in the policy file in JDK 1.4.

          Any unresolved permission is automatically resolved to java.security.UnresolvedPermission and re-resolved later.

          This is actually needed for application defined permission classes where the system cannot load them until the policy has been created, otherwise their static initialization code could take advantage of the lack of security policy at that time.

          Show
          Daniel John Debrunner added a comment - FYI on why the JDK 1.5 specific JMX permissions work in the policy file in JDK 1.4. Any unresolved permission is automatically resolved to java.security.UnresolvedPermission and re-resolved later. This is actually needed for application defined permission classes where the system cannot load them until the policy has been created, otherwise their static initialization code could take advantage of the lack of security policy at that time.
          Hide
          John H. Embretsen added a comment -

          Thanks for useful feedback on the func spec. Attaching a new version (3.1) with the following changes:

          • Clarified that MBean specifics are subject to change.
          • Clarified that this spec assumes that most official documentation will not be available for 10.4.
          • Removed the BuildNumberAsInt attribute of VersionMBean.
          • Removed PropertyPermission from listing in Java Security section. Also improved some of the wording and added "air" to the sample policy file to improve readability.
          • Loosened up some statements in the Management Service section regarding MBeanServers and JMX access which may not be true in all scenarios.
          • Some other cosmetic changes.
          Show
          John H. Embretsen added a comment - Thanks for useful feedback on the func spec. Attaching a new version (3.1) with the following changes: Clarified that MBean specifics are subject to change. Clarified that this spec assumes that most official documentation will not be available for 10.4. Removed the BuildNumberAsInt attribute of VersionMBean. Removed PropertyPermission from listing in Java Security section. Also improved some of the wording and added "air" to the sample policy file to improve readability. Loosened up some statements in the Management Service section regarding MBeanServers and JMX access which may not be true in all scenarios. Some other cosmetic changes.
          Hide
          Daniel John Debrunner added a comment -

          Nice update on the spec:

          • as a general comment I think the spec should focus on the general jmx setup rather than specific MBeans, attributes, operations etc. Maybe just add a sentence indicating the MBeans listed are valid when the spec is written but may be subject to change. E.g. if a new attribute is added to VersionMBean then having to update the spec seems over the top.
          • For VersionMBean I think the getBuildNumberAsInt should be removed, the build number may not be an integer so not providing a conversion method may reinforce that fact. It also adds little value.
          • In the security section it says "When using a custom policy file, the following additional permissions must be granted to derby.jar ". That's not true, Derby could provide monitoring with a sub-set of those permissions. Not sure what should be put here, maybe just a description of why each permission is required.
            IN the same section PropertyPermission should be removed, it's not required for JMX at the moment and may never be.
          • In the MBeanServer section it says:
            "When Derby's Management Service has been started, a valid JMX user will be able to access both the Derby-specific MBeans registered by the Management Service, as well as any other MBeans or MXBeans made available by the managed JVM, if the JMX client supports this."
            It's more complicated than that, given the definition of JMX user, it's not true that such a JMX user "will be able to access both ...", a JMX User may be restricted into which MBeans it can access or even see.
          Show
          Daniel John Debrunner added a comment - Nice update on the spec: as a general comment I think the spec should focus on the general jmx setup rather than specific MBeans, attributes, operations etc. Maybe just add a sentence indicating the MBeans listed are valid when the spec is written but may be subject to change. E.g. if a new attribute is added to VersionMBean then having to update the spec seems over the top. For VersionMBean I think the getBuildNumberAsInt should be removed, the build number may not be an integer so not providing a conversion method may reinforce that fact. It also adds little value. In the security section it says "When using a custom policy file, the following additional permissions must be granted to derby.jar ". That's not true, Derby could provide monitoring with a sub-set of those permissions. Not sure what should be put here, maybe just a description of why each permission is required. IN the same section PropertyPermission should be removed, it's not required for JMX at the moment and may never be. In the MBeanServer section it says: "When Derby's Management Service has been started, a valid JMX user will be able to access both the Derby-specific MBeans registered by the Management Service, as well as any other MBeans or MXBeans made available by the managed JVM, if the JMX client supports this." It's more complicated than that, given the definition of JMX user, it's not true that such a JMX user "will be able to access both ...", a JMX User may be restricted into which MBeans it can access or even see.
          Hide
          John H. Embretsen added a comment -

          Attaching a new version (3.0) of the jmxFuncspec (HTML). I hope I have managed to bring the spec more in alignment with the current state of these features, and what seems to be realistic to include for 10.4. I have also tried to clarify a few issues that were unclear in previous versions of the spec.

          There are still a few minor differences between this spec and what has been committed. The following code changes should be considered:

          VersionMBean:

          • change attribute MaintVersion to MaintenanceVersion

          NetworkServerMBean:

          • change return type of attribute DerbyDrdaStreamOutBufferSize from
            String to int

          ...and some Javadoc issues already mentioned in relevant Jiras.

          Show
          John H. Embretsen added a comment - Attaching a new version (3.0) of the jmxFuncspec (HTML). I hope I have managed to bring the spec more in alignment with the current state of these features, and what seems to be realistic to include for 10.4. I have also tried to clarify a few issues that were unclear in previous versions of the spec. There are still a few minor differences between this spec and what has been committed. The following code changes should be considered: VersionMBean: change attribute MaintVersion to MaintenanceVersion NetworkServerMBean: change return type of attribute DerbyDrdaStreamOutBufferSize from String to int ...and some Javadoc issues already mentioned in relevant Jiras.
          Hide
          John H. Embretsen added a comment -

          Regarding the latest additions to the wiki page http://wiki.apache.org/db-derby/JMXSecurityExpectations:
          (new attempt to comment)

          • JMX Security setups:
          • As a description of how things work, it looks correct to me.
          • Access Control Proposal:
          • See separate mail thread on derby-dev.
          Show
          John H. Embretsen added a comment - Regarding the latest additions to the wiki page http://wiki.apache.org/db-derby/JMXSecurityExpectations: (new attempt to comment) JMX Security setups: As a description of how things work, it looks correct to me. Access Control Proposal: See separate mail thread on derby-dev.
          Hide
          Daniel John Debrunner added a comment -

          Patch jmxPolicyFileChanges_v1.diff committed (revision 629575) - Thanks John.

          Show
          Daniel John Debrunner added a comment - Patch jmxPolicyFileChanges_v1.diff committed (revision 629575) - Thanks John.
          Hide
          John H. Embretsen added a comment -

          I see... I have only tried remote clients with authentication disabled (so far), and in that case all clients can access all MBeans including enabling/disabling the management service when the permissions in the patch are present. I'm no Java security expert, but I'm guessing no Subject/Principal checking is being performed in that case. With JMX authentication enabled additional or different permissions may be required, I guess.

          Thanks for the wiki update - I'll read it more carefully tomorrow

          Show
          John H. Embretsen added a comment - I see... I have only tried remote clients with authentication disabled (so far), and in that case all clients can access all MBeans including enabling/disabling the management service when the permissions in the patch are present. I'm no Java security expert, but I'm guessing no Subject/Principal checking is being performed in that case. With JMX authentication enabled additional or different permissions may be required, I guess. Thanks for the wiki update - I'll read it more carefully tomorrow
          Hide
          Daniel John Debrunner added a comment -

          John> By "Users" I basically meant Network Server VM administrators.

          If that means the "VM-Admin" (from the JMXSecurityExpectations terminology) then that role is the same as a local jmx-client.

          Show
          Daniel John Debrunner added a comment - John> By "Users" I basically meant Network Server VM administrators. If that means the "VM-Admin" (from the JMXSecurityExpectations terminology) then that role is the same as a local jmx-client.
          Hide
          Daniel John Debrunner added a comment -

          With remote JMX clients and JMX authentication then each JMXPrincipal needs permissions to perform any JMX actions (such as finding an MBean, getting an attribute or invoking an operation). The policy file does not grant those permissions and I don't believe it should. I'm just trying to understand what situations you expect it to work.

          I've been updating the wiki page with my experiments of running jmx in various configurations.

          http://wiki.apache.org/db-derby/JMXSecurityExpectations

          I don't think I've tried remote clients with no jmx authentication though, with the patch can such clients do anything useful?

          Show
          Daniel John Debrunner added a comment - With remote JMX clients and JMX authentication then each JMXPrincipal needs permissions to perform any JMX actions (such as finding an MBean, getting an attribute or invoking an operation). The policy file does not grant those permissions and I don't believe it should. I'm just trying to understand what situations you expect it to work. I've been updating the wiki page with my experiments of running jmx in various configurations. http://wiki.apache.org/db-derby/JMXSecurityExpectations I don't think I've tried remote clients with no jmx authentication though, with the patch can such clients do anything useful?
          Hide
          John H. Embretsen added a comment -

          By "Users" I basically meant Network Server VM administrators. As far as I know, the permissions allow Derby's code to instantiate the MBeanServer if needed and register, unregister and be the source of MBeans. I think there is strictly no concept of users, especially no distinction between local and remote users, in this context..., though the source (codebase) of the actions may make a difference. Is there a possibility that such permissions are not needed?

          Show
          John H. Embretsen added a comment - By "Users" I basically meant Network Server VM administrators. As far as I know, the permissions allow Derby's code to instantiate the MBeanServer if needed and register, unregister and be the source of MBeans. I think there is strictly no concept of users, especially no distinction between local and remote users, in this context... , though the source (codebase) of the actions may make a difference. Is there a possibility that such permissions are not needed?
          Hide
          Daniel John Debrunner added a comment -

          By "Users" do you mean local jmx clients, because that's all the permissions will allow isn't it?

          Or are remote jmx-clients allowed if there is no JMX authentication?

          Show
          Daniel John Debrunner added a comment - By "Users" do you mean local jmx clients, because that's all the permissions will allow isn't it? Or are remote jmx-clients allowed if there is no JMX authentication?
          Hide
          John H. Embretsen added a comment -

          The permissions allow out-of-the box monitoring and management of Derby when the Network Server installs its own security manager by default and derby.system.jmx=true. Users can enjoy the JMX functionality without having to resort to the often cumbersome task of customizing a policy file to replace the default. In a shorter perspective, the patch allows Derby developers and testers to try out the JMX functionality in a convenient way. The permissions may have to be tweaked as the JMX functionality changes, though.

          I think the separate subpackages idea, d), is a good option.

          Show
          John H. Embretsen added a comment - The permissions allow out-of-the box monitoring and management of Derby when the Network Server installs its own security manager by default and derby.system.jmx=true. Users can enjoy the JMX functionality without having to resort to the often cumbersome task of customizing a policy file to replace the default. In a shorter perspective, the patch allows Derby developers and testers to try out the JMX functionality in a convenient way. The permissions may have to be tweaked as the JMX functionality changes, though. I think the separate subpackages idea, d), is a good option.
          Hide
          Daniel John Debrunner added a comment -

          John, for the permissions patch, what are your expectations for its purpose, ie. what functionality does it bring?

          Show
          Daniel John Debrunner added a comment - John, for the permissions patch, what are your expectations for its purpose, ie. what functionality does it bring?
          Hide
          Daniel John Debrunner added a comment -

          How about:

          d) derby jars can have a unique package sealed for that jar, e.g. o.a.d.mbeans.drda, o.a.d.mbeans.tools etc.

          Show
          Daniel John Debrunner added a comment - How about: d) derby jars can have a unique package sealed for that jar, e.g. o.a.d.mbeans.drda, o.a.d.mbeans.tools etc.
          Hide
          John H. Embretsen added a comment -

          OK, thanks (I hope you're right regarding Q2 ). I might be able to start progress on DERBY-3435 tomorrow.

          Show
          John H. Embretsen added a comment - OK, thanks (I hope you're right regarding Q2 ). I might be able to start progress on DERBY-3435 tomorrow.
          Hide
          Daniel John Debrunner added a comment -

          Not sure if we are on a) or b) at the moment, depends on where DERBY-3435 goes.

          I don't think there's an issue with jdk 1.5. We have existing code that is specific to jdk 1.5/jse 6.0. Adding more is not an issue.

          Show
          Daniel John Debrunner added a comment - Not sure if we are on a) or b) at the moment, depends on where DERBY-3435 goes. I don't think there's an issue with jdk 1.5. We have existing code that is specific to jdk 1.5/jse 6.0. Adding more is not an issue.
          Hide
          John H. Embretsen added a comment -

          I have a couple of questions related to the building and packaging of MBeans:

          1) All MBean interfaces are placed in the package org.apache.derby.mbeans. All the current classes in this package are included in derby.jar. If someone creates MBeans that are designed to manage or monitor a part of derby that is in a different jar file, e.g. the Network Server (derbynet.jar) or IJ (derbytools.jar), we have a number of options:

          a) Keep the interface in the same jar as the class(es) referencing the interface. This means that MBean interfaces in org.apache.derby.mbeans may be placed in a number of different jar files.
          b) Keep all MBean interfaces in one jar, namely derby.jar.
          c) Include the interface in both derby.jar and the jar which code references the interface.

          I believe a) is the default today. For example, adding a NetworkServer MBean will result in its interface being included in derbynet.jar only. The org.apache.derby.mbeans package would have to be "unsealed".

          Is this the way to proceed, or should we rather pursue option b) or c)?

          2) MBean implementations such as Version and JDBC are compiled with jdk1.4 today. This means that those classes cannot reference the JMX API directly. I wouldn't be surprised if we at some point would like to utilize the JMX notification scheme with some MBeans, or include various MBean metadata to improve usability. This might require access to the JMX API from such classes.
          I would suspect that changing this (to build such MBean implementations with jdk1.5) will require solving a few non-trivial build dependency issues. Is this a valid concern, or do I worry too much?

          Show
          John H. Embretsen added a comment - I have a couple of questions related to the building and packaging of MBeans: 1) All MBean interfaces are placed in the package org.apache.derby.mbeans. All the current classes in this package are included in derby.jar. If someone creates MBeans that are designed to manage or monitor a part of derby that is in a different jar file, e.g. the Network Server (derbynet.jar) or IJ (derbytools.jar), we have a number of options: a) Keep the interface in the same jar as the class(es) referencing the interface. This means that MBean interfaces in org.apache.derby.mbeans may be placed in a number of different jar files. b) Keep all MBean interfaces in one jar, namely derby.jar. c) Include the interface in both derby.jar and the jar which code references the interface. I believe a) is the default today. For example, adding a NetworkServer MBean will result in its interface being included in derbynet.jar only. The org.apache.derby.mbeans package would have to be "unsealed". Is this the way to proceed, or should we rather pursue option b) or c)? 2) MBean implementations such as Version and JDBC are compiled with jdk1.4 today. This means that those classes cannot reference the JMX API directly. I wouldn't be surprised if we at some point would like to utilize the JMX notification scheme with some MBeans, or include various MBean metadata to improve usability. This might require access to the JMX API from such classes. I would suspect that changing this (to build such MBean implementations with jdk1.5) will require solving a few non-trivial build dependency issues. Is this a valid concern, or do I worry too much?
          Hide
          John H. Embretsen added a comment -

          Yes, the permissions seem to work (i.e. not cause failures) on jdk1.4 as well, just as with the old patch (sorry for not replying sooner).. Not sure why it works, though.
          I have not run the full regression test suites with it yet.

          Show
          John H. Embretsen added a comment - Yes, the permissions seem to work (i.e. not cause failures) on jdk1.4 as well, just as with the old patch (sorry for not replying sooner).. Not sure why it works, though. I have not run the full regression test suites with it yet.
          Hide
          Daniel John Debrunner added a comment -

          Thanks for the permissions patch John, does it work on JDK 1.4 since these permissions were only added in JDK 1.5?

          Show
          Daniel John Debrunner added a comment - Thanks for the permissions patch John, does it work on JDK 1.4 since these permissions were only added in JDK 1.5?
          Hide
          John H. Embretsen added a comment -

          Attached a patch implementing a set of default permissions for JMX in the server's policy file in case this is interesting to commit at this point (makes it easier to test run the server with JMX enabled), namely (for derby.jar):

          permission javax.management.MBeanServerPermission "createMBeanServer";
          permission javax.management.MBeanPermission org.apache.derby.*#[org.apache.derby:*]","registerMBean,unregisterMBean";
          permission javax.management.MBeanTrustPermission "register";

          This is a slightly modified variant of the permissions outlined in the comment
          https://issues.apache.org/jira/browse/DERBY-1387?focusedCommentId=12568329#action_12568329
          and works with the current trunk (revision 628812).

          Show
          John H. Embretsen added a comment - Attached a patch implementing a set of default permissions for JMX in the server's policy file in case this is interesting to commit at this point (makes it easier to test run the server with JMX enabled), namely (for derby.jar): permission javax.management.MBeanServerPermission "createMBeanServer"; permission javax.management.MBeanPermission org.apache.derby.*# [org.apache.derby:*] ","registerMBean,unregisterMBean"; permission javax.management.MBeanTrustPermission "register"; This is a slightly modified variant of the permissions outlined in the comment https://issues.apache.org/jira/browse/DERBY-1387?focusedCommentId=12568329#action_12568329 and works with the current trunk (revision 628812).
          Hide
          Daniel John Debrunner added a comment -

          A1) Yes Object instead of ObjectName to allow the code to be compiled and used with JDK 1.4. Also it allows flexibility in any implementation to use a different object to manage its mbeans.

          A2,A3) - will fix

          A4) - yes.

          Show
          Daniel John Debrunner added a comment - A1) Yes Object instead of ObjectName to allow the code to be compiled and used with JDK 1.4. Also it allows flexibility in any implementation to use a different object to manage its mbeans. A2,A3) - will fix A4) - yes.
          Hide
          John H. Embretsen added a comment -

          Some comments regarding (mainly) JMXManagementService.java:

          1) Just to be clear: Is the reason why the unregisterMBean() method takes an
          Object instead of ObjectName as parameter that it should be possible to
          call this method from code compiled with jdk1.4? (Same goes for the
          return type of registerMBean)

          2) The following javadoc comment for registerMBean() now seems outdated:

          • The object name instance
          • represented by the given String will be created by this method.

          3) I think the current use of the term "attribute" in registerMBean() has
          potential to confuse people. In JMX land, the term "attribute" denotes the
          MBean properties that are exposed through getter and setter methods. In
          JMXManagementService.java, the term "attribute" or "nameAttribute" is also
          used for the key properties of an ObjectName, for example "type=Version"
          or "jar=derby.jar".
          I think using a different term for the latter will make it easier for
          other people to get acquainted with this code. Official JMX documentation
          (e.g. the ObjectName javadocs) refers to these properties as
          "Key properties", an unordered set of keys and associated values, so
          perhaps it is better to use that term instead?

          4) Is it true that Monitor.getSystemModule(Module.JMX) will never return null,
          if we keep the current implementation?

          Show
          John H. Embretsen added a comment - Some comments regarding (mainly) JMXManagementService.java: 1) Just to be clear: Is the reason why the unregisterMBean() method takes an Object instead of ObjectName as parameter that it should be possible to call this method from code compiled with jdk1.4? (Same goes for the return type of registerMBean) 2) The following javadoc comment for registerMBean() now seems outdated: The object name instance represented by the given String will be created by this method. 3) I think the current use of the term "attribute" in registerMBean() has potential to confuse people. In JMX land, the term "attribute" denotes the MBean properties that are exposed through getter and setter methods. In JMXManagementService.java, the term "attribute" or "nameAttribute" is also used for the key properties of an ObjectName, for example "type=Version" or "jar=derby.jar". I think using a different term for the latter will make it easier for other people to get acquainted with this code. Official JMX documentation (e.g. the ObjectName javadocs) refers to these properties as "Key properties", an unordered set of keys and associated values, so perhaps it is better to use that term instead? 4) Is it true that Monitor.getSystemModule(Module.JMX) will never return null, if we keep the current implementation?
          Hide
          John H. Embretsen added a comment -

          I created a new wiki page to keep track of MBeans and other related information that is scattered on the wiki and elsewhere:

          http://wiki.apache.org/db-derby/DerbyJMX

          Show
          John H. Embretsen added a comment - I created a new wiki page to keep track of MBeans and other related information that is scattered on the wiki and elsewhere: http://wiki.apache.org/db-derby/DerbyJMX
          Hide
          Daniel John Debrunner added a comment -

          I added a new JDBCMBean that required exposing methods to register and unregister mbeans on the ManagementService api.
          It's created by the jdbc driver (InternalDriver) Revision: 627877.

          It has some attributes and one operation that maps to Driver.acceptsURL.

          Show
          Daniel John Debrunner added a comment - I added a new JDBCMBean that required exposing methods to register and unregister mbeans on the ManagementService api. It's created by the jdbc driver (InternalDriver) Revision: 627877. It has some attributes and one operation that maps to Driver.acceptsURL.
          Hide
          Daniel John Debrunner added a comment -

          Revision: 627826 fixes the unsupported major.minor version in derby.log. The properties in modules.properties were messed up for the jdk requirement. Thanks for noticing the problem John.

          Show
          Daniel John Debrunner added a comment - Revision: 627826 fixes the unsupported major.minor version in derby.log. The properties in modules.properties were messed up for the jdk requirement. Thanks for noticing the problem John.
          Hide
          Daniel John Debrunner added a comment -

          The Unsupported major.minor version 49.0 should not be happening, the monitor is meant not meant to attempt to load the class since it indicates its modules.properties entry indicates jdk 1.5. I'll look to see where I messed up.

          Show
          Daniel John Debrunner added a comment - The Unsupported major.minor version 49.0 should not be happening, the monitor is meant not meant to attempt to load the class since it indicates its modules.properties entry indicates jdk 1.5. I'll look to see where I messed up.
          Hide
          John H. Embretsen added a comment -

          Thanks for the new revision of the patch. The committed code seems like a usable starting point for further work. I hope we're able to add some more useful functionality to it before the 10.4 release - perhaps we could start with a set of monitoring features that are not as sensitive security-wise as some of the ones currently in the funcSpec.

          Due to the missing security permissions Derby's JMX feature does not currently work "out of the box" with the Network Server when using default settings (Security Manager). Although we'll add a set of required permissions to the default policy later, I am slightly worried that no hints about missing permissions are given to the user.

          When I now run the Network Server with jdk1.4 and JMX enabled, I see the following in derby.log:

          2008-02-14 13:40:44.874 GMT Thread[main,5,main] Class org.apache.derby.impl.services.jmx.JMXManagementService java.lang.UnsupportedClassVersionError: org/apache/derby/impl/services/jmx/JMXManagementService (Unsupported major.minor version 49.0), module ignored.

          The message makes sense to me, but will it make sense to the users?

          Also, the fact that the JMXManagementSerrvice now wraps the MBeans in a StandardMBean has some implications that we (and any Derby MBean developers) should be aware of. For example, emitting notifications or implementing MBeanRegistration won't work without some extra effort; see [1]. I think using StandardMBean (which technically makes our MBeans "dynamic" instead of "standard") is useful and provides more flexibility, but I just wanted to point out that it is not necessarily always straight forward.

          [1]: http://blogs.sun.com/jmxetc/entry/javax_management_standardmbean_when_and

          Show
          John H. Embretsen added a comment - Thanks for the new revision of the patch. The committed code seems like a usable starting point for further work. I hope we're able to add some more useful functionality to it before the 10.4 release - perhaps we could start with a set of monitoring features that are not as sensitive security-wise as some of the ones currently in the funcSpec. Due to the missing security permissions Derby's JMX feature does not currently work "out of the box" with the Network Server when using default settings (Security Manager). Although we'll add a set of required permissions to the default policy later, I am slightly worried that no hints about missing permissions are given to the user. When I now run the Network Server with jdk1.4 and JMX enabled, I see the following in derby.log: 2008-02-14 13:40:44.874 GMT Thread [main,5,main] Class org.apache.derby.impl.services.jmx.JMXManagementService java.lang.UnsupportedClassVersionError: org/apache/derby/impl/services/jmx/JMXManagementService (Unsupported major.minor version 49.0), module ignored. The message makes sense to me, but will it make sense to the users? Also, the fact that the JMXManagementSerrvice now wraps the MBeans in a StandardMBean has some implications that we (and any Derby MBean developers) should be aware of. For example, emitting notifications or implementing MBeanRegistration won't work without some extra effort; see [1] . I think using StandardMBean (which technically makes our MBeans "dynamic" instead of "standard") is useful and provides more flexibility, but I just wanted to point out that it is not necessarily always straight forward. [1] : http://blogs.sun.com/jmxetc/entry/javax_management_standardmbean_when_and
          Hide
          Daniel John Debrunner added a comment -

          The functional spec states a set of permissions needed when running under the security manager, but it's a much broader scope of permissions than are actually needed. Derby's documentation (and its default policy files) should be specifying the minimum permissions.

          I think this will be something like:

          permission javax.management.MBeanServerPermission "createMBeanServer";

          permission javax.management.MBeanPermission "org.apache.derby.*#[org.apache.derby:]","register,unregister";

          permission javax.management.MBeanTrustPermission "register"

          The value of the first field in MBeanPermission may need some tweaking, this was just from reading the javadoc.

          Show
          Daniel John Debrunner added a comment - The functional spec states a set of permissions needed when running under the security manager, but it's a much broader scope of permissions than are actually needed. Derby's documentation (and its default policy files) should be specifying the minimum permissions. I think this will be something like: permission javax.management.MBeanServerPermission "createMBeanServer"; permission javax.management.MBeanPermission "org.apache.derby.*# [org.apache.derby:] ","register,unregister"; permission javax.management.MBeanTrustPermission "register" The value of the first field in MBeanPermission may need some tweaking, this was just from reading the javadoc.
          Hide
          Daniel John Debrunner added a comment -

          Committed the simple_9_1 patch and a follow on change that moves the implementation of any Derby MBean out of the o.a.d.mbeans package. Since o.a.d.mbeans becomes part of the public api, having the implementation of beans in there causes issues as some beans will need to be created using implementation specific classes (such as BasicDatabase) which must not appear in javadoc.

          Show
          Daniel John Debrunner added a comment - Committed the simple_9_1 patch and a follow on change that moves the implementation of any Derby MBean out of the o.a.d.mbeans package. Since o.a.d.mbeans becomes part of the public api, having the implementation of beans in there causes issues as some beans will need to be created using implementation specific classes (such as BasicDatabase) which must not appear in javadoc.
          Hide
          Daniel John Debrunner added a comment -

          > My (badly worded) point was rather that the FS mentions
          > org.apache.derby.impl.services.mbeans.BasicManagementService.java
          > in the case where JMX is available, but nothing about
          > org.apache.derby.impl.services.mbeans.NoManagementService
          > in the opposite case

          Ahhh, wasn't expecting those classes to be in the functional spec, since they are implementation details.

          I changed these names (from patch 9 & hence the functional spec) in the simple patch:

          o.a.d.jmx -> o.a.d.mbeans (public api)

          o.a.d.iapi.services.mbeans -> o.a.d.iapi.services.jmx
          o.a.d.impl.services.mbeans -> o.a.d.iapi.services.jmx

          Rationale is that the derby internal code is really dealing with jmx and the public api is exposing mbeans.

          BasicManagementService -> JMXManagementService

          Rationale is that the name gives a better indication of its function (never did like the BasicXXX style in Derby)

          jmxnone.NoManagementService - is a an none functional implementation of (Derby's internal) ManagementService api. It's there to support environments that do not have JMX. Implementing it as an empty class makes the management service module use the existing functionality for handling different environments rather than having special code in the monitor.

          Show
          Daniel John Debrunner added a comment - > My (badly worded) point was rather that the FS mentions > org.apache.derby.impl.services.mbeans.BasicManagementService.java > in the case where JMX is available, but nothing about > org.apache.derby.impl.services.mbeans.NoManagementService > in the opposite case Ahhh, wasn't expecting those classes to be in the functional spec, since they are implementation details. I changed these names (from patch 9 & hence the functional spec) in the simple patch: o.a.d.jmx -> o.a.d.mbeans (public api) o.a.d.iapi.services.mbeans -> o.a.d.iapi.services.jmx o.a.d.impl.services.mbeans -> o.a.d.iapi.services.jmx Rationale is that the derby internal code is really dealing with jmx and the public api is exposing mbeans. BasicManagementService -> JMXManagementService Rationale is that the name gives a better indication of its function (never did like the BasicXXX style in Derby) jmxnone.NoManagementService - is a an none functional implementation of (Derby's internal) ManagementService api. It's there to support environments that do not have JMX. Implementing it as an empty class makes the management service module use the existing functionality for handling different environments rather than having special code in the monitor.
          Hide
          Thomas Nielsen added a comment -

          djd> The Derby interface class ManagementService is an internal api for Derby only, it is not exposed through any jmx interface, only the mbeans are.

          My (badly worded) point was rather that the FS mentions
          org.apache.derby.impl.services.mbeans.BasicManagementService.java
          in the case where JMX is available, but nothing about
          org.apache.derby.impl.services.mbeans.NoManagementService
          in the opposite case

          Show
          Thomas Nielsen added a comment - djd> The Derby interface class ManagementService is an internal api for Derby only, it is not exposed through any jmx interface, only the mbeans are. My (badly worded) point was rather that the FS mentions org.apache.derby.impl.services.mbeans.BasicManagementService.java in the case where JMX is available, but nothing about org.apache.derby.impl.services.mbeans.NoManagementService in the opposite case
          Hide
          Daniel John Debrunner added a comment -

          > - Whenever started on a non-JMX capable platform, the funcspec says "Derby will start with the Management Service disabled." The patch still has a service called "jmxnone". This service should probably be mentioned in the Agent Level section along with the services that are there when JMX is available.

          The jmxnone implementation is how the management service is disabled. The Derby interface class ManagementService is an internal api for Derby only, it is not exposed through any jmx interface, only the mbeans are.

          > - Version MBean:
          > o Alpha should probably be renamed to isAlpha to align with isBeta?

          Agreed there is some cleanup needed there, I didn't understand why there is a getIsAlpha and an isAlpha().

          Show
          Daniel John Debrunner added a comment - > - Whenever started on a non-JMX capable platform, the funcspec says "Derby will start with the Management Service disabled." The patch still has a service called "jmxnone". This service should probably be mentioned in the Agent Level section along with the services that are there when JMX is available. The jmxnone implementation is how the management service is disabled. The Derby interface class ManagementService is an internal api for Derby only, it is not exposed through any jmx interface, only the mbeans are. > - Version MBean: > o Alpha should probably be renamed to isAlpha to align with isBeta? Agreed there is some cleanup needed there, I didn't understand why there is a getIsAlpha and an isAlpha().
          Hide
          Thomas Nielsen added a comment -

          Before reading the complete jira, I did a quick, unbiased read of the FuncSpec v2.1 and had a peek at the simplified v9 patch Dan posted, and put together some minor comments:

          • Whenever started on a non-JMX capable platform, the funcspec says "Derby will start with the Management Service disabled." The patch still has a service called "jmxnone". This service should probably be mentioned in the Agent Level section along with the services that are there when JMX is available.
          • Using the Platform MBeanserver and Standard MBeans is the way to go. To my knowledge using these will also cater for future MXBeans. Good choices!
          • Version MBean:
            o Alpha should probably be renamed to isAlpha to align with isBeta?
            o MaintVersion should probably be written out to MaintenanceVersion?
          • MDatabaseMBean:
            o The first 'M' seems to break the naming scheme used for the other MBeans? No explaination for this deviation is given in the FS. Leftover from the Commons Modeller "Model MBean" that was once rejected?
            o addDBUser() will delete a user too according to the FS, so the name should probably be changed to manageDBUser() or similar?

          On skimming the complete Jira afterwards, I see some of my initial comments on the MDatabaseMBean have already been raised by others.

          Show
          Thomas Nielsen added a comment - Before reading the complete jira, I did a quick, unbiased read of the FuncSpec v2.1 and had a peek at the simplified v9 patch Dan posted, and put together some minor comments: Whenever started on a non-JMX capable platform, the funcspec says "Derby will start with the Management Service disabled." The patch still has a service called "jmxnone". This service should probably be mentioned in the Agent Level section along with the services that are there when JMX is available. Using the Platform MBeanserver and Standard MBeans is the way to go. To my knowledge using these will also cater for future MXBeans. Good choices! Version MBean: o Alpha should probably be renamed to isAlpha to align with isBeta? o MaintVersion should probably be written out to MaintenanceVersion? MDatabaseMBean: o The first 'M' seems to break the naming scheme used for the other MBeans? No explaination for this deviation is given in the FS. Leftover from the Commons Modeller "Model MBean" that was once rejected? o addDBUser() will delete a user too according to the FS, so the name should probably be changed to manageDBUser() or similar? On skimming the complete Jira afterwards, I see some of my initial comments on the MDatabaseMBean have already been raised by others.
          Hide
          Daniel John Debrunner added a comment -

          Simplified version of the full patch (version 9) to allow commit of initial functionality without having to solve all the security & other issues. In addition to the changes listed in an earlier comment this patch also unregisters the mbeans it has registered on derby system shutdown. I don't think the old patch was doing that.

          Show
          Daniel John Debrunner added a comment - Simplified version of the full patch (version 9) to allow commit of initial functionality without having to solve all the security & other issues. In addition to the changes listed in an earlier comment this patch also unregisters the mbeans it has registered on derby system shutdown. I don't think the old patch was doing that.
          Hide
          John H. Embretsen added a comment -

          Dan wrote:
          > I've been working on a much simplified initial jmx setup based upon patch 9, I'll attach it as a patch sometime tomorrow once I've checked it out some more. (...)

          That certainly sounds great - from the description it seems like a good approach to me. I probably won't be able to take a look at it until the middle of the week, though, but don't let that stop you.

          Show
          John H. Embretsen added a comment - Dan wrote: > I've been working on a much simplified initial jmx setup based upon patch 9, I'll attach it as a patch sometime tomorrow once I've checked it out some more. (...) That certainly sounds great - from the description it seems like a good approach to me. I probably won't be able to take a look at it until the middle of the week, though, but don't let that stop you.
          Hide
          Daniel John Debrunner added a comment -

          I've been working on a much simplified initial jmx setup based upon patch 9, I'll attach it as a patch sometime tomorrow once I've checked it out some more.

          This is aimed at getting the ball rolling (incremental development) rather than trying to solve all the security issues etc. with one big patch.

          The patch is different from patch 9 in these ways:

          • renaming packages in services to o.a.d. {iapi,impl}

            .jmx

          • MBeans in o.a.d.mbeans
          • only support a single Version bean (no security issues) [others can be added later]
          • renamed BasicManagementService to JMXManagementService
          • much simplified implementation of ManagementService, single method registerMBean. This of course can be expanded later.
          • simplified code within ManagementService implementation and implements the module api's correctly, also uses Java 5 generics
          • jdk14 booting implemented by standard modules.properties techniques, no need for special code in the monitor
          • no changes to any policy file - needs to be a follow on patch

          If this seems an acceptable approach then I'll commit it within a couple of days.

          Show
          Daniel John Debrunner added a comment - I've been working on a much simplified initial jmx setup based upon patch 9, I'll attach it as a patch sometime tomorrow once I've checked it out some more. This is aimed at getting the ball rolling (incremental development) rather than trying to solve all the security issues etc. with one big patch. The patch is different from patch 9 in these ways: renaming packages in services to o.a.d. {iapi,impl} .jmx MBeans in o.a.d.mbeans only support a single Version bean (no security issues) [others can be added later] renamed BasicManagementService to JMXManagementService much simplified implementation of ManagementService, single method registerMBean. This of course can be expanded later. simplified code within ManagementService implementation and implements the module api's correctly, also uses Java 5 generics jdk14 booting implemented by standard modules.properties techniques, no need for special code in the monitor no changes to any policy file - needs to be a follow on patch If this seems an acceptable approach then I'll commit it within a couple of days.
          Hide
          John H. Embretsen added a comment -

          I'd like to take a step back and try to clarify and write down what the community's expectations are with regard to the security architecture of the JMX features proposed by this Jira issue.

          I started a wiki page at

          http://wiki.apache.org/db-derby/JMXSecurityExpectations

          to do this. If those who are interested would like to take a look that would be great. If you think the page is useful, we may use it as basis for designing a security architecture for JMX in Derby. If you don't think it's useful, perhaps it was nevertheless a good exercise for me to do this, in order to wrap my head around a few of the things we might need to deal with (it would be nice if someone could tell me if I am way off about something on that page).
          Please speak up if you have comments.

          Show
          John H. Embretsen added a comment - I'd like to take a step back and try to clarify and write down what the community's expectations are with regard to the security architecture of the JMX features proposed by this Jira issue. I started a wiki page at http://wiki.apache.org/db-derby/JMXSecurityExpectations to do this. If those who are interested would like to take a look that would be great. If you think the page is useful, we may use it as basis for designing a security architecture for JMX in Derby. If you don't think it's useful, perhaps it was nevertheless a good exercise for me to do this, in order to wrap my head around a few of the things we might need to deal with (it would be nice if someone could tell me if I am way off about something on that page). Please speak up if you have comments.
          Hide
          Daniel John Debrunner added a comment -

          For the security issue one idea is consider is packaging Derby's mbeans in a separate jar file (e.g. derbyjmx.jar).
          This jar could be automatically in the class path of derby.jar but having it separate would allow it to be granted different permissions.

          If the mbeans are in derby.jar then they will most likely have the permission to read Derby's system properties (derby.*),
          putting them in a separate jar allows some control to some administrator (vm-admin?), thus they could have the option of:

          • not allowing any jmx access to system properties while continuing to use the other administration facilities of Derby's mbeans
          • allowing only read access to derby's system properties (that are exposed through Derby's mbeans)
          • allowing read/write access to derby's system properties (that are exposed through Derby's mbeans)

          > I noticed Dan's suggestion of including a restricted/limited set of MBeans/attributes/operations in the first version of this feature

          My suggestion was slightly different, a first patch that gets working mbeans with limited attributes and operations to allow folks to play with them.
          Then any amount of added new functionality before a release that contains mbeans (ie. 10.4), the only requirement would be some agreement that new functionality does not expose security holes. E.g. a initial patch of working functionality could be:

          • version mbean - as it is today
          • system mbean - maybe any system property read-only that does not expose security information (e.g. LocksWaitTimeOut ok, DatabaseFullAccessUsers not ok)
          • network server mbean - ping only
          • database mbean - name, locale, read-only
          Show
          Daniel John Debrunner added a comment - For the security issue one idea is consider is packaging Derby's mbeans in a separate jar file (e.g. derbyjmx.jar). This jar could be automatically in the class path of derby.jar but having it separate would allow it to be granted different permissions. If the mbeans are in derby.jar then they will most likely have the permission to read Derby's system properties (derby.*), putting them in a separate jar allows some control to some administrator (vm-admin?), thus they could have the option of: not allowing any jmx access to system properties while continuing to use the other administration facilities of Derby's mbeans allowing only read access to derby's system properties (that are exposed through Derby's mbeans) allowing read/write access to derby's system properties (that are exposed through Derby's mbeans) > I noticed Dan's suggestion of including a restricted/limited set of MBeans/attributes/operations in the first version of this feature My suggestion was slightly different, a first patch that gets working mbeans with limited attributes and operations to allow folks to play with them. Then any amount of added new functionality before a release that contains mbeans (ie. 10.4), the only requirement would be some agreement that new functionality does not expose security holes. E.g. a initial patch of working functionality could be: version mbean - as it is today system mbean - maybe any system property read-only that does not expose security information (e.g. LocksWaitTimeOut ok, DatabaseFullAccessUsers not ok) network server mbean - ping only database mbean - name, locale, read-only
          Hide
          John H. Embretsen added a comment -

          @Rick: The rather unhelpful error messages you were seeing are mostly caused by JConsole not having access to the classes required to unwind the message of the root cause Exception. Bernt mentioned a work-around for that: https://issues.apache.org/jira/browse/DERBY-1387?focusedCommentId=12537011#action_12537011
          Perhaps it is possible to improve this if the MBeans are implemented a little differently, I think I saw something about that on the web somewhere (might require using different MBean types or something, I'll see if I can find a reference).

          The work-around works for all errors except the one you get when shutting down the server. If the server VM exits when you shutdown the server (true if you're using NetworkServerControl or derbyrun.jar on the command line), then the MBeanServer will also exit, so the connection between JConsole and the server is broken abruptly. Not sure if this is possible to fix for generic JMX clients such as JConsole.

          @All - Security:

          So, it seems clear to me from the recent comments that we need to do something about the security issues. I think we can agree on the following:

          A valid JMX user (a user able to connect via JMX to Derby's MBeanServer) should not be able to access information or perform operations that would otherwise be restricted by Derby's existing security mechanisms (authentication, authorization, Security Manager, etc.).

          I noticed Dan's suggestion of including a restricted/limited set of MBeans/attributes/operations in the first version of this feature, which seems sensible to me. Other ideas are of course welcome.

          Show
          John H. Embretsen added a comment - @Rick: The rather unhelpful error messages you were seeing are mostly caused by JConsole not having access to the classes required to unwind the message of the root cause Exception. Bernt mentioned a work-around for that: https://issues.apache.org/jira/browse/DERBY-1387?focusedCommentId=12537011#action_12537011 Perhaps it is possible to improve this if the MBeans are implemented a little differently, I think I saw something about that on the web somewhere (might require using different MBean types or something, I'll see if I can find a reference). The work-around works for all errors except the one you get when shutting down the server. If the server VM exits when you shutdown the server (true if you're using NetworkServerControl or derbyrun.jar on the command line), then the MBeanServer will also exit, so the connection between JConsole and the server is broken abruptly. Not sure if this is possible to fix for generic JMX clients such as JConsole. @All - Security: So, it seems clear to me from the recent comments that we need to do something about the security issues. I think we can agree on the following: A valid JMX user (a user able to connect via JMX to Derby's MBeanServer) should not be able to access information or perform operations that would otherwise be restricted by Derby's existing security mechanisms (authentication, authorization, Security Manager, etc.). I noticed Dan's suggestion of including a restricted/limited set of MBeans/attributes/operations in the first version of this feature, which seems sensible to me. Other ideas are of course welcome.
          Hide
          Daniel John Debrunner added a comment -

          I can see the argument for disallowing Derby's jmx beans (by setting derby.system.jmx=false) and probably it should be an opt-in system (ie. mbeans off by default).

          Having derby.system.jmx being static though is a huge pain, seems that most useful would be the ability to enable mbeans against a running system.

          Also having the ability at the database level to control its DatabaseMBean would seem useful (i.e. a database can indicate if it will register a DatabaseMBean or not).

          I don't think any of these are essential for this work, but it may be something to think about, especially around the api of enabling system or database mbeans. A single boolean property may be too inflexible and cause confusion if a finer-grained scheme is added later.

          One thought is to have a simple mbean that controls if other mbeans are enabled. This meta-mbean would require system authentication to enable other mbeans (e.g. a property or operation to start the system mbean). Probably requires the mbean security model to be worked out first.

          Show
          Daniel John Debrunner added a comment - I can see the argument for disallowing Derby's jmx beans (by setting derby.system.jmx=false) and probably it should be an opt-in system (ie. mbeans off by default). Having derby.system.jmx being static though is a huge pain, seems that most useful would be the ability to enable mbeans against a running system. Also having the ability at the database level to control its DatabaseMBean would seem useful (i.e. a database can indicate if it will register a DatabaseMBean or not). I don't think any of these are essential for this work, but it may be something to think about, especially around the api of enabling system or database mbeans. A single boolean property may be too inflexible and cause confusion if a finer-grained scheme is added later. One thought is to have a simple mbean that controls if other mbeans are enabled. This meta-mbean would require system authentication to enable other mbeans (e.g. a property or operation to start the system mbean). Probably requires the mbean security model to be worked out first.
          Hide
          Rick Hillegas added a comment -

          I believe the reason that I was not able to connect at the end of my experiment was this: the server was actually brought down. Again, without presenting credentials, this seems like the wrong behavior to me.

          Show
          Rick Hillegas added a comment - I believe the reason that I was not able to connect at the end of my experiment was this: the server was actually brought down. Again, without presenting credentials, this seems like the wrong behavior to me.
          Hide
          Rick Hillegas added a comment -

          Thanks for the updated spec, John. I brought up a Derby server with the JMX beans enabled and I ran JConsole against it. Here are some initial reactions:

          1) I was pleased that the Sun JMX properties worked with my Apple VM.

          2) I brought up the server with authentication enabled but I didn't have to supply credentials in order to view the NetworkServer and System MBeans. That seems wrong to me. The Version MBean was also viewable but that seems ok to me.

          3) I was able to change properties in the NetworkServer MBean--at least my changes were accepted on the screen. However, when I tried to change properties in the System MBean, I was slapped with an "access denied (java.util.PropertyPermission * read,write)" diagnostic.

          4) Using the System MBean, it seemed that I was able to boot an existing database and then access its MBean. Again, without supplying credentials, this seemed wrong to me. Most of the properties were listed as "Unavailable".

          5) Using the System MBean, I tried to boot a nonexistent database. This gave me the following diagnostic:

          Problem invoking bootDatabase: java.rmi.UnmarshalException: Error unmarshaling return: nested exception is:
          java.lang.ClassNotFoundException: org.apache.derby.impl.jdbc.EmbedSQLException (no security manager: RMI class loader disabled)

          6) I saw the same diagnostic when I used the System MBean to boot an encrypted database.

          7) I saw a similar diagnostic when I tried to create a user via the MBean for the existing database.

          8) When I invoked the shutdown operation on the System MBean, I saw the following diagnostic:

          Problem invoking shutdown: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
          java.io.EOFException

          9) However, after dismissing the diagnostic windows, the MBeans disappeared from the screen and I was presented with a blank JConsole window with only a "Connection" menu item in the menu bar. However, I was unable to connect via that menu item.

          For the record, this is the startup command which I used to boot the server:

          java -Dderby.connection.requireAuthentication=true \
          -Dderby.authentication.provider=BUILTIN \
          -Dderby.user.\"rick\"=rickspassword \
          -Dderby.database.sqlAuthorization=true \
          -Dcom.sun.management.jmxremote.port=1528 \
          -Dcom.sun.management.jmxremote.authenticate=false \
          -Dcom.sun.management.jmxremote.ssl=false \
          -Dderby.system.jmx=true \
          org.apache.derby.drda.NetworkServerControl start -p 8246

          and this is the command which I used to boot JConsole:

          jconsole 1735

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Thanks for the updated spec, John. I brought up a Derby server with the JMX beans enabled and I ran JConsole against it. Here are some initial reactions: 1) I was pleased that the Sun JMX properties worked with my Apple VM. 2) I brought up the server with authentication enabled but I didn't have to supply credentials in order to view the NetworkServer and System MBeans. That seems wrong to me. The Version MBean was also viewable but that seems ok to me. 3) I was able to change properties in the NetworkServer MBean--at least my changes were accepted on the screen. However, when I tried to change properties in the System MBean, I was slapped with an "access denied (java.util.PropertyPermission * read,write)" diagnostic. 4) Using the System MBean, it seemed that I was able to boot an existing database and then access its MBean. Again, without supplying credentials, this seemed wrong to me. Most of the properties were listed as "Unavailable". 5) Using the System MBean, I tried to boot a nonexistent database. This gave me the following diagnostic: Problem invoking bootDatabase: java.rmi.UnmarshalException: Error unmarshaling return: nested exception is: java.lang.ClassNotFoundException: org.apache.derby.impl.jdbc.EmbedSQLException (no security manager: RMI class loader disabled) 6) I saw the same diagnostic when I used the System MBean to boot an encrypted database. 7) I saw a similar diagnostic when I tried to create a user via the MBean for the existing database. 8) When I invoked the shutdown operation on the System MBean, I saw the following diagnostic: Problem invoking shutdown: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.io.EOFException 9) However, after dismissing the diagnostic windows, the MBeans disappeared from the screen and I was presented with a blank JConsole window with only a "Connection" menu item in the menu bar. However, I was unable to connect via that menu item. For the record, this is the startup command which I used to boot the server: java -Dderby.connection.requireAuthentication=true \ -Dderby.authentication.provider=BUILTIN \ -Dderby.user.\"rick\"=rickspassword \ -Dderby.database.sqlAuthorization=true \ -Dcom.sun.management.jmxremote.port=1528 \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -Dderby.system.jmx=true \ org.apache.derby.drda.NetworkServerControl start -p 8246 and this is the command which I used to boot JConsole: jconsole 1735 Thanks, -Rick
          Hide
          Daniel John Debrunner added a comment -

          The approach to security for the database mbean concerns me:

          • The authenticateUser approach exposes a huge security hole where any authenticated jmx user can perform dbo operations for the database even if they do not have dbo credentials. I think this is unacceptable.
          • It is defining a security model that does not match existing jmx approaches (e.g. the jmx tutorial describes three security models including two that address fine grained authorization (which is the issue here)).

          I wonder if it is worth splitting the patch in two logical steps.

          1) Adding the framework to support Derby mbeans and skeleton beans. E.g. add a database bean that only exposes limited information, such as the database id and possibly the name.

          2) Add functionality to the beans in a secure manner as required.

          This suits the model of incremental development and allows others to get involved in adding new information to existing beans or adding new beans in the framework.

          Show
          Daniel John Debrunner added a comment - The approach to security for the database mbean concerns me: The authenticateUser approach exposes a huge security hole where any authenticated jmx user can perform dbo operations for the database even if they do not have dbo credentials. I think this is unacceptable. It is defining a security model that does not match existing jmx approaches (e.g. the jmx tutorial describes three security models including two that address fine grained authorization (which is the issue here)). I wonder if it is worth splitting the patch in two logical steps. 1) Adding the framework to support Derby mbeans and skeleton beans. E.g. add a database bean that only exposes limited information, such as the database id and possibly the name. 2) Add functionality to the beans in a secure manner as required. This suits the model of incremental development and allows others to get involved in adding new information to existing beans or adding new beans in the framework.
          Hide
          Rick Hillegas added a comment -

          On September 27 Bernt shared his opinion on the topic of whether to add a special knob which enables the Derby JMX beans. He was in favor of adding the knob. For him, the problem case was a shrink-wrapped embedded application which wants to use JMX as its configuration tool but which doesn't want the user to monkey with the shrink-wrapped database.

          Show
          Rick Hillegas added a comment - On September 27 Bernt shared his opinion on the topic of whether to add a special knob which enables the Derby JMX beans. He was in favor of adding the knob. For him, the problem case was a shrink-wrapped embedded application which wants to use JMX as its configuration tool but which doesn't want the user to monkey with the shrink-wrapped database.
          Hide
          John H. Embretsen added a comment -

          @Kim: Thanks. As the spec stabilizes, it will probably become more clear what is needed wrt. documentation.

          @Thomas: I agree that it could be clearer where the "home" of each part of derby's public API is. The current public API is not very clear in this regard either, see e.g. http://db.apache.org/derby/javadoc/publishedapi/jdbc3/, but I could try to clarify the jmx package(s) somewhat in this funcspec. As part of the documentation effort, we should try to describe this in the package-summary of the new package(s) and in other related Javadoc.

          @Dan:

          I suppose I was influenced by the current implementation when I specified a different package for the Network Server MBean. This MBean was moved to the drda package because of some packaging issue (according to comments in Jira), but if keeping all MBeans in the same package is practically feasible I'm all for it. Since the MBeans are not directly related to JDBC I'm reluctant to putting the others in the o.a.d.jdbc package.

          SystemMBean:
          I see the point about exposing the derby.system.home property to JMX clients, but I don't expect this to be a big issue, since when you enable JMX monitoring of your JVM, you usually have access to lots of file system paths anyway (library path, boot class path, etc.). One use case I think is valid is the need to read this property in a setting where you are running Derby via third-party tools (IDEs, application servers, db management tools etc.), and you don't necessarily know the working directory of the VM running the Derby system, or if derby.system.home has been set automatically by such a tool. One way to find out where your databases (specified with relative paths) end up on your file system is to check the value of derby.system.home.

          MDatabaseMBean:
          Not sure what the M stands for, to be honest. I was guessing "Managed Database", but could be wrong. Perhaps the M should be removed unless someone can shed some light on this.
          SystemHome in MDatabaseMBean shouldn't be there. It's a bug in the funcSpec - it's not available in the current implementation (patch 9).
          The AuthenticatedAsUser attribute could use a better description, for sure. It returns the user name of the user who has been authenticated through JMX, i.e. the last valid user name supplied as parameter to the authenticateAsUser() operation. If no user is authenticated through JMX, the empty string is returned.

          General:

          I'll update the funcspec later this week. I intend to incorporate the above feedback, but I also think some details are needed wrt. MBean naming and (ObjectName) types and server domains.

          Regarding the security issue mentioned in recent comments:

          It could be argued that when you enable JMX management and monitoring you explicitly expose your system for such access, and that if you want to control this you need to enable JMX authentication (there is out-of-the-box support for the use of password files and SSL, see http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html ).

          I noticed, however, a note about a potential security vulnerability wrt. JMX password authentication with J2SE 5.0 (see the above link, look for "WARNING"). A bit more work (mimicking out-of-the-box management) is required for us to work around this...

          I came across a blog post [1] which has some details around this (see its comments), and which includes example use cases wrt. JMX authentication and authorization, which may be of help if we don't decide to settle with the defaults. I've just skimmed through it so far, but perhaps someone familiar with the current and soon-finished security features of Derby could take a look at [1] and see if some kind of integration with existing features is desired and feasible?

          For the short run, however, I propose simply going with the defaults, as long as we are clear about the risks.

          [1]: http://blogs.sun.com/lmalventosa/entry/jmx_authentication_authorization

          Show
          John H. Embretsen added a comment - @Kim: Thanks. As the spec stabilizes, it will probably become more clear what is needed wrt. documentation. @Thomas: I agree that it could be clearer where the "home" of each part of derby's public API is. The current public API is not very clear in this regard either, see e.g. http://db.apache.org/derby/javadoc/publishedapi/jdbc3/ , but I could try to clarify the jmx package(s) somewhat in this funcspec. As part of the documentation effort, we should try to describe this in the package-summary of the new package(s) and in other related Javadoc. @Dan: I suppose I was influenced by the current implementation when I specified a different package for the Network Server MBean. This MBean was moved to the drda package because of some packaging issue (according to comments in Jira), but if keeping all MBeans in the same package is practically feasible I'm all for it. Since the MBeans are not directly related to JDBC I'm reluctant to putting the others in the o.a.d.jdbc package. SystemMBean: I see the point about exposing the derby.system.home property to JMX clients, but I don't expect this to be a big issue, since when you enable JMX monitoring of your JVM, you usually have access to lots of file system paths anyway (library path, boot class path, etc.). One use case I think is valid is the need to read this property in a setting where you are running Derby via third-party tools (IDEs, application servers, db management tools etc.), and you don't necessarily know the working directory of the VM running the Derby system, or if derby.system.home has been set automatically by such a tool. One way to find out where your databases (specified with relative paths) end up on your file system is to check the value of derby.system.home. MDatabaseMBean: Not sure what the M stands for, to be honest. I was guessing "Managed Database", but could be wrong. Perhaps the M should be removed unless someone can shed some light on this. SystemHome in MDatabaseMBean shouldn't be there. It's a bug in the funcSpec - it's not available in the current implementation (patch 9). The AuthenticatedAsUser attribute could use a better description, for sure. It returns the user name of the user who has been authenticated through JMX, i.e. the last valid user name supplied as parameter to the authenticateAsUser() operation. If no user is authenticated through JMX, the empty string is returned. General: I'll update the funcspec later this week. I intend to incorporate the above feedback, but I also think some details are needed wrt. MBean naming and (ObjectName) types and server domains. Regarding the security issue mentioned in recent comments: It could be argued that when you enable JMX management and monitoring you explicitly expose your system for such access, and that if you want to control this you need to enable JMX authentication (there is out-of-the-box support for the use of password files and SSL, see http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html ). I noticed, however, a note about a potential security vulnerability wrt. JMX password authentication with J2SE 5.0 (see the above link, look for "WARNING"). A bit more work (mimicking out-of-the-box management) is required for us to work around this... I came across a blog post [1] which has some details around this (see its comments), and which includes example use cases wrt. JMX authentication and authorization, which may be of help if we don't decide to settle with the defaults. I've just skimmed through it so far, but perhaps someone familiar with the current and soon-finished security features of Derby could take a look at [1] and see if some kind of integration with existing features is desired and feasible? For the short run, however, I propose simply going with the defaults, as long as we are clear about the risks. [1] : http://blogs.sun.com/lmalventosa/entry/jmx_authentication_authorization
          Hide
          Daniel John Debrunner added a comment -

          I'm a little lost on the logic on not putting all the MBeans in the same package, isn't the o.a.d.jmx package now the jmx package for Derby? Seems logical that all the mbeans should be there. Otherwise why not just put the "embedded" mbeans in o.a.d.jdbc?

          SystemMBean
          Is there a good use case for having SystemHome as a property. Secure systems tend to not reveal information about the file system structure, less information is good.

          MDatabaseMBean
          What does the M mean in 'MDatabase'?

          Why have SystemHome in MDatabaseMBean, if it's going to be available then it should only be there through SystemMBean?

          What does the property AuthenticatedAsUser represent, the table just says look at the operation but no deatails are given there?

          ps. ignore my comment about duplicate version files, I think my search for MBean in the patch turned into a search & replace.

          Show
          Daniel John Debrunner added a comment - I'm a little lost on the logic on not putting all the MBeans in the same package, isn't the o.a.d.jmx package now the jmx package for Derby? Seems logical that all the mbeans should be there. Otherwise why not just put the "embedded" mbeans in o.a.d.jdbc? SystemMBean Is there a good use case for having SystemHome as a property. Secure systems tend to not reveal information about the file system structure, less information is good. MDatabaseMBean What does the M mean in 'MDatabase'? Why have SystemHome in MDatabaseMBean, if it's going to be available then it should only be there through SystemMBean? What does the property AuthenticatedAsUser represent, the table just says look at the operation but no deatails are given there? ps. ignore my comment about duplicate version files, I think my search for MBean in the patch turned into a search & replace.
          Hide
          Kim Haase added a comment -

          This revised spec is excellent, John, from the perspective of documentation. All the added detail is very helpful.

          Show
          Kim Haase added a comment - This revised spec is excellent, John, from the perspective of documentation. All the added detail is very helpful.
          Hide
          Thomas Nielsen added a comment -

          John> Attached new funcSpec (jmxFuncspec.html), version 2.1, ... Please review.

          A small comment:
          Since the funcSpec only considers the server-side JMX functionallity it would probably be good to rename the section "Public API" to something that explicitly indicates it is for use in the enigne (server-side) only? I first read this section thinking it was the "Public [Client-side] API".
          I'm afraid I don't have any good suggestions other than "Derby Engine/Server-side JMX API".

          Show
          Thomas Nielsen added a comment - John> Attached new funcSpec (jmxFuncspec.html), version 2.1, ... Please review. A small comment: Since the funcSpec only considers the server-side JMX functionallity it would probably be good to rename the section "Public API" to something that explicitly indicates it is for use in the enigne (server-side) only? I first read this section thinking it was the "Public [Client-side] API". I'm afraid I don't have any good suggestions other than "Derby Engine/Server-side JMX API".
          Hide
          John H. Embretsen added a comment -

          Attached new funcSpec (jmxFuncspec.html), version 2.1, incorporating changes and clarifications mentioned in comments from Dan and myself since February 1. The most significant change is the moving of most of the public MBean interfaces to a new package, org.apache.derby.jmx. The NetworkServerControlMgmtMBean was moved to org.apache.derby.drda. Please review.

          I have also verified that the com.sun.management.jmxremote properties are not part of an official standard, but may be considered de facto standard, and included comments about this in the funcSpec.

          Some implementation TODOs have now accumulated, including:

          • rearranging the package layout as specified in the funcspec, and renaming one of the MBeans.
          • closing the most apparent security holes (such as being able to connect to a db requiring authentication with multiple JMX-clients, without having to provide credentials in more than one of them).
          • supporting/utilizing the Derby Monitor's dependency features (modules.properties).
          • i18n/L10n-enabled error messages/warnings.

          Feel free to reassign this issue if you feel that you may have something to contribute in these or other areas of the implementation, as I intend to spend some time creating tests (DERBY-3385) as well.

          Show
          John H. Embretsen added a comment - Attached new funcSpec (jmxFuncspec.html), version 2.1, incorporating changes and clarifications mentioned in comments from Dan and myself since February 1. The most significant change is the moving of most of the public MBean interfaces to a new package, org.apache.derby.jmx. The NetworkServerControlMgmtMBean was moved to org.apache.derby.drda. Please review. I have also verified that the com.sun.management.jmxremote properties are not part of an official standard, but may be considered de facto standard, and included comments about this in the funcSpec. Some implementation TODOs have now accumulated, including: rearranging the package layout as specified in the funcspec, and renaming one of the MBeans. closing the most apparent security holes (such as being able to connect to a db requiring authentication with multiple JMX-clients, without having to provide credentials in more than one of them). supporting/utilizing the Derby Monitor's dependency features (modules.properties). i18n/L10n-enabled error messages/warnings. Feel free to reassign this issue if you feel that you may have something to contribute in these or other areas of the implementation, as I intend to spend some time creating tests ( DERBY-3385 ) as well.
          Hide
          John H. Embretsen added a comment -

          Strange - I'm not able to see what you were seeing, Dan. Here's some grep results from the patch file DERBY-1387-9.diff that I just downloaded from Jira onto a machine which has not seen it before:

          $grep -c "Index.*Version.java" DERBY-1387-9.diff
          1

          $grep "Version.java" DERBY-1387-9.diff
          Index: java/engine/org/apache/derby/impl/services/mbeans/Version.java
          — java/engine/org/apache/derby/impl/services/mbeans/Version.java (revision 0)
          +++ java/engine/org/apache/derby/impl/services/mbeans/Version.java (revision 0)

          $grep "class Version" DERBY-1387-9.diff
          +public class Version implements VersionMBean {

          With Standard MBeans (which is the type of MBeans included in the patch), the requirement is that there is an interface which name ends with "MBean" and a class implementing that class, with the same name but without the "MBean" ending. As far as I know there may be other rules for other types of MBeans (Dynamic, Model, ...). As far as I can see the patch fulfills this requirement, but if I'm misunderstanding something, I need some help to understand exactly what.

          Show
          John H. Embretsen added a comment - Strange - I'm not able to see what you were seeing, Dan. Here's some grep results from the patch file DERBY-1387 -9.diff that I just downloaded from Jira onto a machine which has not seen it before: $grep -c "Index.*Version.java" DERBY-1387 -9.diff 1 $grep "Version.java" DERBY-1387 -9.diff Index: java/engine/org/apache/derby/impl/services/mbeans/Version.java — java/engine/org/apache/derby/impl/services/mbeans/Version.java (revision 0) +++ java/engine/org/apache/derby/impl/services/mbeans/Version.java (revision 0) $grep "class Version" DERBY-1387 -9.diff +public class Version implements VersionMBean { With Standard MBeans (which is the type of MBeans included in the patch), the requirement is that there is an interface which name ends with "MBean" and a class implementing that class, with the same name but without the "MBean" ending. As far as I know there may be other rules for other types of MBeans (Dynamic, Model, ...). As far as I can see the patch fulfills this requirement, but if I'm misunderstanding something, I need some help to understand exactly what.
          Hide
          Daniel John Debrunner added a comment -

          The patch seems to contain two Version.java's, one an interface and one a class in the same package.

          Index: java/engine/org/apache/derby/impl/services/mbeans/Version.java
          Index: java/engine/org/apache/derby/impl/services/mbeans/Version.java

          public class Version implements Version

          not sure how this is possible ....

          With MBeans is there a requirement for an interface and an implementation class?

          Show
          Daniel John Debrunner added a comment - The patch seems to contain two Version.java's, one an interface and one a class in the same package. Index: java/engine/org/apache/derby/impl/services/mbeans/Version.java Index: java/engine/org/apache/derby/impl/services/mbeans/Version.java public class Version implements Version not sure how this is possible .... With MBeans is there a requirement for an interface and an implementation class?
          Hide
          John H. Embretsen added a comment -

          Thanks for taking a look at this, Dan - the more eyes on this the better.

          > 1) Is a derby.system.jmx needed? What would be the downside of always having the jmx beans available, or available if the jvm has jmx running?
          > This may depend on security issues, I haven't looked at that in detail yet.

          Potential new and unknown security risks is my primary concern at the moment, weighing in against enabling JMX by default. I believe that a portion of the security risks introduced with this feature have not been identified and evaluated fully yet. Other than that I have no big concerns about enabling it by default. The performance penalty should be negligible, at least. It should, however, be possible for the paranoid user to disable the Management Service, I think. Then I guess such a property would be needed anyway...?

          > 2) In the "Enabling Management Features" section, are the com.sun properties standard JMX, or are they specific to Sun's implementation of the JVM? useful to state this in the specification, so that any user documentation reflects reality.

          Good question, I've been wondering that myself. I cannot find any reference to those properties in the JMX specs (but maybe I didn't look hard enough). I have seen the com.sun.management.jmxremote properties in used in reference to the newer JVMs from other vendors than Sun, such as IBM and BEA (JRockit), so it seems to be a de facto standard at least...

          > 3) The spec says that classes in org.apache.derby.impl.services.mbeans. will be added to the public api. The 'org.apache.derby.impl.' set of packages is for implementations of internal apis (o.a.d.iapi). If these classes are external then they need to be moved to a new package. o.a.d.jmx?

          I see, makes sense. I think moving the public interfaces to a new package is something that should be considered for the next version of the patch.

          > As a minor point I wouldn't have Derby in the name of these classes.

          That is probably redundant, yes. DerbySystemMBean should be renamed to SystemMBean in the next version of the patch.

          > Can authenticateAsUser(String user, String password) in the database bean be explained more fully?
          > Is there a single database bean for a database, or is a new one created for each jmx session or connection (not sure of correct term here)?

          There is currently a single MDatabaseMBean per database.

          > If there is a single bean for a database then this authenticateAsUser seems to open up a big security hole, once this operation is made any other valid jmx user can reconfigure the database, even if they don't have valid database permissions.

          Correct, that is one of the issues I had in mind when I noted that the current patch "lacks a few essential security measures".

          > Or can the authentication information be limited to a single jmx session, even with a single bean?

          Not with the current implementation, but perhaps it is possible to implement some kind of session handling to support something like this? Not sure if the previous contributors to this feature have given this any thought...

          > Also, why is authenticateAsUser limited to the BUILTIN authentication scheme, since it is just providing user/password to a connection request won't it be independent of the authentication scheme in effect?

          Right, I believe this is a cut&paste error in the funcspec. All it does is to establish a regular JDBC connection using DriverManager. I'll rectify it in the next version. Thanks for noticing!

          Show
          John H. Embretsen added a comment - Thanks for taking a look at this, Dan - the more eyes on this the better. > 1) Is a derby.system.jmx needed? What would be the downside of always having the jmx beans available, or available if the jvm has jmx running? > This may depend on security issues, I haven't looked at that in detail yet. Potential new and unknown security risks is my primary concern at the moment, weighing in against enabling JMX by default. I believe that a portion of the security risks introduced with this feature have not been identified and evaluated fully yet. Other than that I have no big concerns about enabling it by default. The performance penalty should be negligible, at least. It should, however, be possible for the paranoid user to disable the Management Service, I think. Then I guess such a property would be needed anyway...? > 2) In the "Enabling Management Features" section, are the com.sun properties standard JMX, or are they specific to Sun's implementation of the JVM? useful to state this in the specification, so that any user documentation reflects reality. Good question, I've been wondering that myself. I cannot find any reference to those properties in the JMX specs (but maybe I didn't look hard enough). I have seen the com.sun.management.jmxremote properties in used in reference to the newer JVMs from other vendors than Sun, such as IBM and BEA (JRockit), so it seems to be a de facto standard at least... > 3) The spec says that classes in org.apache.derby.impl.services.mbeans. will be added to the public api. The 'org.apache.derby.impl.' set of packages is for implementations of internal apis (o.a.d.iapi). If these classes are external then they need to be moved to a new package. o.a.d.jmx? I see, makes sense. I think moving the public interfaces to a new package is something that should be considered for the next version of the patch. > As a minor point I wouldn't have Derby in the name of these classes. That is probably redundant, yes. DerbySystemMBean should be renamed to SystemMBean in the next version of the patch. > Can authenticateAsUser(String user, String password) in the database bean be explained more fully? > Is there a single database bean for a database, or is a new one created for each jmx session or connection (not sure of correct term here)? There is currently a single MDatabaseMBean per database. > If there is a single bean for a database then this authenticateAsUser seems to open up a big security hole, once this operation is made any other valid jmx user can reconfigure the database, even if they don't have valid database permissions. Correct, that is one of the issues I had in mind when I noted that the current patch "lacks a few essential security measures". > Or can the authentication information be limited to a single jmx session, even with a single bean? Not with the current implementation, but perhaps it is possible to implement some kind of session handling to support something like this? Not sure if the previous contributors to this feature have given this any thought... > Also, why is authenticateAsUser limited to the BUILTIN authentication scheme, since it is just providing user/password to a connection request won't it be independent of the authentication scheme in effect? Right, I believe this is a cut&paste error in the funcspec. All it does is to establish a regular JDBC connection using DriverManager. I'll rectify it in the next version. Thanks for noticing!
          Hide
          Daniel John Debrunner added a comment - - edited

          Can authenticateAsUser(String user, String password) in the database bean be explained more fully?
          Is there a single database bean for a database, or is a new one created for each jmx session or connection (not sure of correct term here)?

          If there is a single bean for a database then this authenticateAsUser seems to open up a big security hole, once this operation is made any other valid jmx user can reconfigure the database, even if they don't have valid database permissions.

          Or can the authentication information be limited to a single jmx session, even with a single bean?

          Also, why is authenticateAsUser limited to the BUILTIN authentication scheme, since it is just providing user/password to a connection request won't it be independent of the authentication scheme in effect?

          Show
          Daniel John Debrunner added a comment - - edited Can authenticateAsUser(String user, String password) in the database bean be explained more fully? Is there a single database bean for a database, or is a new one created for each jmx session or connection (not sure of correct term here)? If there is a single bean for a database then this authenticateAsUser seems to open up a big security hole, once this operation is made any other valid jmx user can reconfigure the database, even if they don't have valid database permissions. Or can the authentication information be limited to a single jmx session, even with a single bean? Also, why is authenticateAsUser limited to the BUILTIN authentication scheme, since it is just providing user/password to a connection request won't it be independent of the authentication scheme in effect?
          Hide
          Daniel John Debrunner added a comment -

          Thanks for the updated specification: Some initial questions:

          1) Is a derby.system.jmx needed? What would be the downside of always having the jmx beans available, or available if the jvm has jmx running?
          This may depend on security issues, I haven't looked at that in detail yet.

          2) In the "Enabling Management Features" section, are the com.sun properties standard JMX, or are they specific to Sun's implementation of the JVM? useful to state this in the specification, so that any user documentation reflects reality.

          3) The spec says that classes in org.apache.derby.impl.services.mbeans. will be added to the public api. The 'org.apache.derby.impl.' set of packages is for implementations of internal apis (o.a.d.iapi). If these classes are external then they need to be moved to a new package. o.a.d.jmx?
          As a minor point I wouldn't have Derby in the name of these classes.

          Show
          Daniel John Debrunner added a comment - Thanks for the updated specification: Some initial questions: 1) Is a derby.system.jmx needed? What would be the downside of always having the jmx beans available, or available if the jvm has jmx running? This may depend on security issues, I haven't looked at that in detail yet. 2) In the "Enabling Management Features" section, are the com.sun properties standard JMX, or are they specific to Sun's implementation of the JVM? useful to state this in the specification, so that any user documentation reflects reality. 3) The spec says that classes in org.apache.derby.impl.services.mbeans. will be added to the public api. The 'org.apache.derby.impl.' set of packages is for implementations of internal apis (o.a.d.iapi). If these classes are external then they need to be moved to a new package. o.a.d.jmx? As a minor point I wouldn't have Derby in the name of these classes.
          Hide
          John H. Embretsen added a comment -

          Attached a new version of the FuncSpec for this feature, based on my own interpretation of the current implementation and previous comments to this issue.

          There are a few things that were unclear to me, such as whether or not to change the statement "The service acts as a factory which is able to create all the MBeans and provides an API for registering and exposing MBeans.", which currently does not match the implementation (the NetworkServerControlMgmtMBean is the exception). I left it as it were for now.

          I also think more hands and eyes are needed in the Security section, where I've added a couple of paragraphs about authentication. I believe the current patch lacks a few essential security measures, and the current spec is most likely missing out on a few things in that area.

          I have added a section on how to use these new features, so that anyone can download the latest patch and give it a try without too much head-scratching. Comments are always welcome.

          Show
          John H. Embretsen added a comment - Attached a new version of the FuncSpec for this feature, based on my own interpretation of the current implementation and previous comments to this issue. There are a few things that were unclear to me, such as whether or not to change the statement "The service acts as a factory which is able to create all the MBeans and provides an API for registering and exposing MBeans.", which currently does not match the implementation (the NetworkServerControlMgmtMBean is the exception). I left it as it were for now. I also think more hands and eyes are needed in the Security section, where I've added a couple of paragraphs about authentication. I believe the current patch lacks a few essential security measures, and the current spec is most likely missing out on a few things in that area. I have added a section on how to use these new features, so that anyone can download the latest patch and give it a try without too much head-scratching. Comments are always welcome.
          Hide
          Rick Hillegas added a comment -

          Thanks for the new patch, John. A couple comments:

          1) The informational/error messages which return to the user still need to be localized. It's probably a good idea to get this out the way before you get too deep into testing. See Bernt's response on October 8, 2007.

          2) I was pleasantly surprised to see that when I run this under jdk1.4, the security manager doesn't choke on the missing jmx permissions. I guess the vm must be silently swallowing ClassNotFoundExceptions.

          3) I'm looking forward to some basic tests which will show me how to model some experiments. In particular, I want to see how this JMX-initiated shutdown plays with the new permissions introduced by DERBY-2109.

          Show
          Rick Hillegas added a comment - Thanks for the new patch, John. A couple comments: 1) The informational/error messages which return to the user still need to be localized. It's probably a good idea to get this out the way before you get too deep into testing. See Bernt's response on October 8, 2007. 2) I was pleasantly surprised to see that when I run this under jdk1.4, the security manager doesn't choke on the missing jmx permissions. I guess the vm must be silently swallowing ClassNotFoundExceptions. 3) I'm looking forward to some basic tests which will show me how to model some experiments. In particular, I want to see how this JMX-initiated shutdown plays with the new permissions introduced by DERBY-2109 .
          Hide
          John H. Embretsen added a comment -

          Dan wrote:

          > Why is this needed? Derby supports loading modules defined in modules.properties and that has a
          > mechanism to indicate that a module requires a specific jvm or set of Java classes. Is there some
          > reason a new mechanism is being invented?

          Well... kindof. The reason is that I tried using modules.properties for this but my brief experiments had no effect. So there's probably a piece of the module puzzle that I have missed, and it might be due to how the JMX module is currently started - I'm not sure, since I didn't write most of this code, and I would most likely need to spend quite some time with a debugger to understand all the details. My goal was to get this up and running so I could start writing tests for it, but if I get some helpful pointers I'll be happy to change how this works.

          Show
          John H. Embretsen added a comment - Dan wrote: > Why is this needed? Derby supports loading modules defined in modules.properties and that has a > mechanism to indicate that a module requires a specific jvm or set of Java classes. Is there some > reason a new mechanism is being invented? Well... kindof. The reason is that I tried using modules.properties for this but my brief experiments had no effect. So there's probably a piece of the module puzzle that I have missed, and it might be due to how the JMX module is currently started - I'm not sure, since I didn't write most of this code, and I would most likely need to spend quite some time with a debugger to understand all the details. My goal was to get this up and running so I could start writing tests for it, but if I get some helpful pointers I'll be happy to change how this works.
          Hide
          Daniel John Debrunner added a comment -

          >> - added a check in FileMonitor for the presence of certain JMX classes
          before allowing JMX to be enabled. A warning is printed to derby.log and
          (if relevant) the Network Server console if JMX support is not available
          but derby.system.jmx=true. This probably needs some polishing, at least
          in the FileMonitor, where I've simply called report() with a hard-coded
          String as parameter.

          Why is this needed? Derby supports loading modules defined in modules.properties and that has a mechanism to indicate that a module requires a specific jvm or set of Java classes. Is there some reason a new mechanism is being invented?

          Show
          Daniel John Debrunner added a comment - >> - added a check in FileMonitor for the presence of certain JMX classes before allowing JMX to be enabled. A warning is printed to derby.log and (if relevant) the Network Server console if JMX support is not available but derby.system.jmx=true. This probably needs some polishing, at least in the FileMonitor, where I've simply called report() with a hard-coded String as parameter. Why is this needed? Derby supports loading modules defined in modules.properties and that has a mechanism to indicate that a module requires a specific jvm or set of Java classes. Is there some reason a new mechanism is being invented?
          Hide
          John H. Embretsen added a comment -

          I should probably mention that I have run suites.All and derbyall successfully using jdk1.4 with (roughly) the -9 patch (I did some minor tweaking after testing but before submitting), so I'm fairly confident that it won't break existing code in an obvious way. Will run more tests later.

          Show
          John H. Embretsen added a comment - I should probably mention that I have run suites.All and derbyall successfully using jdk1.4 with (roughly) the -9 patch (I did some minor tweaking after testing but before submitting), so I'm fairly confident that it won't break existing code in an obvious way. Will run more tests later.
          Hide
          John H. Embretsen added a comment -

          Uploaded a new patch, DERBY-1387-9.diff + .stat, replacing previous patches.

          Changes since the previous (1387-8) patch:

          • made org.apache.derby.iapi.services.mbeans.ManagementService independent
            of JMX / jdk1.5, as explained in a previous comment. Thus, this patch
            should build just fine without requiring additional libraries, though
            the JMX features are not available in a 1.4 VM.
          • added missing MDatabase class file.
          • Changed o/a/d/impl/services/build.xml's "compile_impl_services_jdk15"
            target to use the default compiler instead of jdk1.6 to compile 1.5 code,
            with java15compile.classpath as the classpath, and included the
            1.5-specific JMX functionality (impl/services/mbeans/*) in this target.
          • added a check in FileMonitor for the presence of certain JMX classes
            before allowing JMX to be enabled. A warning is printed to derby.log and
            (if relevant) the Network Server console if JMX support is not available
            but derby.system.jmx=true. This probably needs some polishing, at least
            in the FileMonitor, where I've simply called report() with a hard-coded
            String as parameter.

          (I may also have missed something somewhere wrt. jdk1.4 since I have
          not done any extensive testing, and I'm not fully on top of how the
          monitor/module system works at this time.)

          • improved Network Server error handling/reporting wrt. JMX
          • resolved patch incompatilbility with current trunk wrt. server.policy
          • added necessary permissions (present in server.policy) to template.policy
          • fixed typo in java/engine/org/apache/derby/impl/services/build.xml
          • removed some spurious tabs from some of the new code segments
          • fixed some Javadoc errors

          There are still some TODOs for this feature, including:

          • proper authentication/authorization/system privileges support?
          • Javadoc for all interfaces (a lot is missing in e.g. MDatabaseMBean.java)
          • better error messages and message handling
          • documentation
          • testing

          There are also a few other things worth mentioning/asking about, such as:

          • why does NetworkServerControlImpl.java create the MBean directly instead
            of using a factory method via the ManagementService interface?
            (contrary to Ole Gunnar's comment of 01/Aug/07 01:30 AM + funcSpec)
            Is it because the MBean requires a reference to NetworkServerControlImpl,
            and the resulting dependencies/build issues are non-trivial to solve?
          • currently, the NetworkServerControlMgmtMBean is in the same package as
            NetworkServerControlImpl, and is compiled with jdk1.4, meaning that
            this MBean cannot reference the JMX API directly (for example: Emit or
            handle Notifications). We may want to change this in a future revision
            of this feature.
          • what is special about BaseMonitor.runWithState()? That is, if the JMX
            module is started differently from other modules, why? I tried adding
            the JMX module with class dependencies to
            org/apache/derby/modules.properties without seeing any effect; could it
            be related to this?
          • when shutting down the server via JMX, no shutdown messages are written
            to the log or the console. If shutdown is supposed to be supported, this
            should be improved.

          Anyway, since this now compiles with the current trunk and seems to run OK
          with all supported JVM versions, I intend to update the functional spec
          and work on testing next.

          Show
          John H. Embretsen added a comment - Uploaded a new patch, DERBY-1387 -9.diff + .stat, replacing previous patches. Changes since the previous (1387-8) patch: made org.apache.derby.iapi.services.mbeans.ManagementService independent of JMX / jdk1.5, as explained in a previous comment. Thus, this patch should build just fine without requiring additional libraries, though the JMX features are not available in a 1.4 VM. added missing MDatabase class file. Changed o/a/d/impl/services/build.xml's "compile_impl_services_jdk15" target to use the default compiler instead of jdk1.6 to compile 1.5 code, with java15compile.classpath as the classpath, and included the 1.5-specific JMX functionality (impl/services/mbeans/*) in this target. added a check in FileMonitor for the presence of certain JMX classes before allowing JMX to be enabled. A warning is printed to derby.log and (if relevant) the Network Server console if JMX support is not available but derby.system.jmx=true. This probably needs some polishing, at least in the FileMonitor, where I've simply called report() with a hard-coded String as parameter. (I may also have missed something somewhere wrt. jdk1.4 since I have not done any extensive testing, and I'm not fully on top of how the monitor/module system works at this time.) improved Network Server error handling/reporting wrt. JMX resolved patch incompatilbility with current trunk wrt. server.policy added necessary permissions (present in server.policy) to template.policy fixed typo in java/engine/org/apache/derby/impl/services/build.xml removed some spurious tabs from some of the new code segments fixed some Javadoc errors There are still some TODOs for this feature, including: proper authentication/authorization/system privileges support? Javadoc for all interfaces (a lot is missing in e.g. MDatabaseMBean.java) better error messages and message handling documentation testing There are also a few other things worth mentioning/asking about, such as: why does NetworkServerControlImpl.java create the MBean directly instead of using a factory method via the ManagementService interface? (contrary to Ole Gunnar's comment of 01/Aug/07 01:30 AM + funcSpec) Is it because the MBean requires a reference to NetworkServerControlImpl, and the resulting dependencies/build issues are non-trivial to solve? currently, the NetworkServerControlMgmtMBean is in the same package as NetworkServerControlImpl, and is compiled with jdk1.4, meaning that this MBean cannot reference the JMX API directly (for example: Emit or handle Notifications). We may want to change this in a future revision of this feature. what is special about BaseMonitor.runWithState()? That is, if the JMX module is started differently from other modules, why? I tried adding the JMX module with class dependencies to org/apache/derby/modules.properties without seeing any effect; could it be related to this? when shutting down the server via JMX, no shutdown messages are written to the log or the console. If shutdown is supposed to be supported, this should be improved. Anyway, since this now compiles with the current trunk and seems to run OK with all supported JVM versions, I intend to update the functional spec and work on testing next.
          Hide
          John H. Embretsen added a comment -

          Hi Rick,

          Regarding the scenarios 1) and 2) you outlined wrt. users trying to enable JMX on jdk1.4, I have (for now) solved this by adding checks in the monitor for the availability of the classes java.lang.management.ManagementFactory and javax.management.ObjectName. The latter is part of the reference implementation, but the former is not, and both are currently required by Derby's JMX functionality.

          If any of these classes are not present in the classpath, a warning is written to derby.log, and Derby boots normally without JMX enabled. In addition, there is a check in NetworkServerControl, writing a warning to the server console if the JMX module is not available (jdk1.4). Exactly what these warning messages should say and how to issue them may need some tweaking/polishing, I presume...

          I will upload a new patch shortly, including these changes. I'll also try to clarify the (lack of) jdk1.4 support in an update to the functional spec.

          Show
          John H. Embretsen added a comment - Hi Rick, Regarding the scenarios 1) and 2) you outlined wrt. users trying to enable JMX on jdk1.4, I have (for now) solved this by adding checks in the monitor for the availability of the classes java.lang.management.ManagementFactory and javax.management.ObjectName. The latter is part of the reference implementation, but the former is not, and both are currently required by Derby's JMX functionality. If any of these classes are not present in the classpath, a warning is written to derby.log, and Derby boots normally without JMX enabled. In addition, there is a check in NetworkServerControl, writing a warning to the server console if the JMX module is not available (jdk1.4). Exactly what these warning messages should say and how to issue them may need some tweaking/polishing, I presume... I will upload a new patch shortly, including these changes. I'll also try to clarify the (lack of) jdk1.4 support in an update to the functional spec.
          Hide
          Rick Hillegas added a comment -

          Hi John,

          It sounds as though you are on a good track here. It's hard to say without seeing the revised patch. Some more defensive code may be needed, however. I am worried that the following may happen:

          1) The customer runs on jdk1.4 without the extra JMX jar file but nevertheless sets derby.system.jmx=true.

          or

          2) Same as (1) but the extra JMX jar file IS included on the classpath.

          In these cases I'm worried that we will see boot errors when the monitor tries to bring up the ManagementService. In case (1) I'm worried that BasicManagementService will fail to boot because the JMX references can't be resolved. In case (2) the BasicManagementService, compiled into Java 5 byte code, will fail to load on jdk1.4.

          To handle (1), you may want to instrument the monitor with defensive code so that it won't boot the ManagementService unless the MBeanServer appears on the classpath--and emit an easy-to-understand diagnostic. To handle (2) you may want to do one of the following:

          a) Instrument the monitor so that it only boots ManagementService if the VM is at level Java 5 or higher (otherwise emit a friendly diagnostic)

          or

          b) Change the mbeans build target in impl/services/build.xml so that the mbeans package is compiled into 1.4 byte code.

          Show
          Rick Hillegas added a comment - Hi John, It sounds as though you are on a good track here. It's hard to say without seeing the revised patch. Some more defensive code may be needed, however. I am worried that the following may happen: 1) The customer runs on jdk1.4 without the extra JMX jar file but nevertheless sets derby.system.jmx=true. or 2) Same as (1) but the extra JMX jar file IS included on the classpath. In these cases I'm worried that we will see boot errors when the monitor tries to bring up the ManagementService. In case (1) I'm worried that BasicManagementService will fail to boot because the JMX references can't be resolved. In case (2) the BasicManagementService, compiled into Java 5 byte code, will fail to load on jdk1.4. To handle (1), you may want to instrument the monitor with defensive code so that it won't boot the ManagementService unless the MBeanServer appears on the classpath--and emit an easy-to-understand diagnostic. To handle (2) you may want to do one of the following: a) Instrument the monitor so that it only boots ManagementService if the VM is at level Java 5 or higher (otherwise emit a friendly diagnostic) or b) Change the mbeans build target in impl/services/build.xml so that the mbeans package is compiled into 1.4 byte code.
          Hide
          John H. Embretsen added a comment -

          I finally got this patch to build without requiring extra jar files. I have tested that the Network Server runs fine with jdk1.4 if jmx is not enabled, and that basic JMX management works fine when using JDK 6, but I have not run other tests so far. (Trying to run using jdk1.4 with jmx enabled currently results in a fatal error, which is something I hope to improve eventually by providing a more helpful error message)

          I would however appreciate some input on whether my solution so far is something we can live with or if it needs refinements. This is what I did to make it build:

          • Made org.apache.derby.iapi.services.mbeans.ManagementService independent
            of JMX / jdk1.5:

          This was necessary because ManagementService is an interface
          representing the (JMX) module that is referenced elsewhere in
          the Derby code, regardless of the availability of JMX support.

          The implementation(s) of this interface may (must) refer to the JMX
          API, but the interface itself may not. Otherwise, all code referencing
          this interface must also be compiled with JMX support in the
          classpath, which is not the default on jdk1.4. Care should be taken not
          to call methods of this interface if JMX support is not available.

          Changes were:

          • Removed getMBeanServer() method (was not used), which returned a
            JMX object (MBeanServer). Callers (e.g. NetworkServerControlImpl) can
            access (create, destroy, register) MBeans through this interface, but may
            no longer get direct access to the MBean server unless they call the
            implementing class (BasicManagementService) directly.
          • Changed second parameter of registerMBean() method to take a String
            representing a JMX ObjectName instead of the ObjectName itself. It
            is then up to the implementing class to create the actual ObjectName
            object based on that String.
          • NetworkServerControlImpl now calls the ManagementService interface
            without having to refer to the JMX API (e.g. ObjectName), which means it
            may be compiled with jdk1.4 and be runnable without JMX support.
          • Changed build scripts so that org.apache.derby.iapi.services.mbeans is
            compiled with $ {compile.classpath}

            (jdk1.4 / jsr169).

          • Added missing class o.a.d.impl.services.MDatabase.java.

          My JMX experience may be limited, but not more so than my knowledge of Derby code internals I'll continue to work on an updated patch, but any comments or advice will be appreciated in the mean time.

          Show
          John H. Embretsen added a comment - I finally got this patch to build without requiring extra jar files. I have tested that the Network Server runs fine with jdk1.4 if jmx is not enabled, and that basic JMX management works fine when using JDK 6, but I have not run other tests so far. (Trying to run using jdk1.4 with jmx enabled currently results in a fatal error, which is something I hope to improve eventually by providing a more helpful error message) I would however appreciate some input on whether my solution so far is something we can live with or if it needs refinements. This is what I did to make it build: Made org.apache.derby.iapi.services.mbeans.ManagementService independent of JMX / jdk1.5: This was necessary because ManagementService is an interface representing the (JMX) module that is referenced elsewhere in the Derby code, regardless of the availability of JMX support. The implementation(s) of this interface may (must) refer to the JMX API, but the interface itself may not. Otherwise, all code referencing this interface must also be compiled with JMX support in the classpath, which is not the default on jdk1.4. Care should be taken not to call methods of this interface if JMX support is not available. Changes were: Removed getMBeanServer() method (was not used), which returned a JMX object (MBeanServer). Callers (e.g. NetworkServerControlImpl) can access (create, destroy, register) MBeans through this interface, but may no longer get direct access to the MBean server unless they call the implementing class (BasicManagementService) directly. Changed second parameter of registerMBean() method to take a String representing a JMX ObjectName instead of the ObjectName itself. It is then up to the implementing class to create the actual ObjectName object based on that String. NetworkServerControlImpl now calls the ManagementService interface without having to refer to the JMX API (e.g. ObjectName), which means it may be compiled with jdk1.4 and be runnable without JMX support. Changed build scripts so that org.apache.derby.iapi.services.mbeans is compiled with $ {compile.classpath} (jdk1.4 / jsr169). Added missing class o.a.d.impl.services.MDatabase.java. My JMX experience may be limited, but not more so than my knowledge of Derby code internals I'll continue to work on an updated patch, but any comments or advice will be appreciated in the mean time.
          Hide
          John H. Embretsen added a comment -

          By the way, I don't think that the "compile_reference" target is the culprit, even if that is the last target printed by ant. I added some echo statements to trunk/java/engine/org/apache/derby/iapi/reference/build.xml, but that target always completes without issues. You can also see this by looking at the verbose ant output; ant exits iapi/reference/build.xml before the build failure occurs.

          It is after ant returns to the caller, target "compile_iapi_error_jsr169" in the file trunk/java/engine/org/apache/derby/iapi/build.xml, that the error occurs, while executing the javac task in that target. This I don't understand (because there are no direct references to the JMX classes in <include name="$

          {derby.dir}

          /iapi/error/**"/>), but I'll try to fix the other things first, and see if the problem disappears.

          Show
          John H. Embretsen added a comment - By the way, I don't think that the "compile_reference" target is the culprit, even if that is the last target printed by ant. I added some echo statements to trunk/java/engine/org/apache/derby/iapi/reference/build.xml, but that target always completes without issues. You can also see this by looking at the verbose ant output; ant exits iapi/reference/build.xml before the build failure occurs. It is after ant returns to the caller, target "compile_iapi_error_jsr169" in the file trunk/java/engine/org/apache/derby/iapi/build.xml, that the error occurs, while executing the javac task in that target. This I don't understand (because there are no direct references to the JMX classes in <include name="$ {derby.dir} /iapi/error/**"/>), but I'll try to fix the other things first, and see if the problem disappears.
          Hide
          John H. Embretsen added a comment -

          Thanks Rick, I'm not fully up to speed on all the various ant properties used in the build, and the various property files used, so this was very useful. I will take a look at the references you gave and try to eliminate JMX from being compiled with 1.4.

          Show
          John H. Embretsen added a comment - Thanks Rick, I'm not fully up to speed on all the various ant properties used in the build, and the various property files used, so this was very useful. I will take a look at the references you gave and try to eliminate JMX from being compiled with 1.4.
          Hide
          Rick Hillegas added a comment -

          Hi John,

          After applying the patch and running "ant all", I see the following sequence of ant targets being invoked:

          engine

          ...

          compile_iapi_error_jsr169:

          compile_reference:

          The error is coming from compile_reference which is a target in trunk/java/engine/org/apache/derby/iapi/reference/build.xml. Looking at the <javac> task in that target, I see that the classpath is set to be just $

          {compile.classpath}

          , which is the set of jars in the JDK, in this case JDK1.4. So none of the jars in tools/java are being pulled in.

          If you want to just hack this together, then you can do the following:

          1) in tools/ant/extrapath.properties, add a new variable which points at the jmxri.jar file you have added.

          2) wire that variable into the classpath in the <javac> task in compile_reference.

          However, trying to force the JMX support to compile under jdk1.4 has some pitfalls to it. It either involves:

          a) asking people to download another (optional) jar file--this at a time when we are trying to simplify the build and eliminate the optional jar files

          b) find or construct a freeware version of the JMX jar file (perhaps by mining the Harmony project?)

          I would recommend compiling the JMX support under Java 5 instead, where you will be guaranteed to have the JMX libraries. You can look at trunk/java/testing/org/apache/derby/build.xml for an example of how to do this. I think it would be ok to say that the JMX support has only been tested under Java 5 and that we do not recommend its use on previous VMs--if necessary, you could put in some code to prevent the JMX support from running on old VMs.

          Show
          Rick Hillegas added a comment - Hi John, After applying the patch and running "ant all", I see the following sequence of ant targets being invoked: engine ... compile_iapi_error_jsr169: compile_reference: The error is coming from compile_reference which is a target in trunk/java/engine/org/apache/derby/iapi/reference/build.xml. Looking at the <javac> task in that target, I see that the classpath is set to be just $ {compile.classpath} , which is the set of jars in the JDK, in this case JDK1.4. So none of the jars in tools/java are being pulled in. If you want to just hack this together, then you can do the following: 1) in tools/ant/extrapath.properties, add a new variable which points at the jmxri.jar file you have added. 2) wire that variable into the classpath in the <javac> task in compile_reference. However, trying to force the JMX support to compile under jdk1.4 has some pitfalls to it. It either involves: a) asking people to download another (optional) jar file--this at a time when we are trying to simplify the build and eliminate the optional jar files b) find or construct a freeware version of the JMX jar file (perhaps by mining the Harmony project?) I would recommend compiling the JMX support under Java 5 instead, where you will be guaranteed to have the JMX libraries. You can look at trunk/java/testing/org/apache/derby/build.xml for an example of how to do this. I think it would be ok to say that the JMX support has only been tested under Java 5 and that we do not recommend its use on previous VMs--if necessary, you could put in some code to prevent the JMX support from running on old VMs.
          Hide
          John H. Embretsen added a comment -

          It turns out that the patch builds fine with the revision for which the original patch (v8) was made, but not with an updated sandbox. I tracked it down to having to do with revision 605224, which enables automatic setting of ant properties (DERBY-3117). This is probably an issue because I have j14lib set in my ant.properties file, because the property setter uses this value to determine the classpath used during compilation. If I set the property dontAutosetProps=true in ant.properties it builds fine with recent trunks as well, but only as long as jmxri.jar is in tools/java.

          I'll ask about this issue (that java14compile.classpath from compilepath.properties is ignored if j14lib is set) in a separate thread on derby-dev.

          Now, I could use some tips for how to set up the build system so that we can build the JMX support with the jdk1.5 or 1.6 compiler (which has included support for JMX) and not require jmxri.jar. The naive version of me assumes that this would be doable by editing a couple of build.xml files, but given that I don't even understand why jmxri.jar seems to be needed for compiling classes in o.a.d/iapi/error, I'm sure some advice would be useful.

          Show
          John H. Embretsen added a comment - It turns out that the patch builds fine with the revision for which the original patch (v8) was made, but not with an updated sandbox. I tracked it down to having to do with revision 605224, which enables automatic setting of ant properties ( DERBY-3117 ). This is probably an issue because I have j14lib set in my ant.properties file, because the property setter uses this value to determine the classpath used during compilation. If I set the property dontAutosetProps=true in ant.properties it builds fine with recent trunks as well, but only as long as jmxri.jar is in tools/java. I'll ask about this issue (that java14compile.classpath from compilepath.properties is ignored if j14lib is set) in a separate thread on derby-dev. Now, I could use some tips for how to set up the build system so that we can build the JMX support with the jdk1.5 or 1.6 compiler (which has included support for JMX) and not require jmxri.jar. The naive version of me assumes that this would be doable by editing a couple of build.xml files, but given that I don't even understand why jmxri.jar seems to be needed for compiling classes in o.a.d/iapi/error, I'm sure some advice would be useful.
          Hide
          John H. Embretsen added a comment - - edited

          I am trying to get an updated version of the latest patch to compile, but I'm having the same problem Rick reported on Nov 20. After putting some echo statements into some of the build.xml files and running with verbose and debug flags, I still don't know why it happens.

          I have added jmxri.jar to tools/java as Bernt mentioned, but it doesn't help (it doesn't seem like it's being picked up into the compile classpath anyway). In any case, I have no idea why ant is reporting errors for org/apache/derby/iapi/services/mbeans/ManagementService when it is supposed to compile files in org/apache/derby/iapi/error/ only, as part of the compile_iapi_error_jsr169 target in java/engine/org/apache/derby/iapi/reference/build.xml.

          From what I can tell, the patch does not touch any files in org/apache/derby/iapi/error/.

          Any ideas?

          (...)
          [echo] Done with compile_reference
          [ant] Exiting /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/reference/build.xml.
          [echo] First javac in compile_iapi_error_jsr169
          [echo] compile.classpath is: /usr/local/java/jdk1.4/jre/lib/charsets.jar:/usr/local/java/jdk1.4/jre/lib/jce.jar:/usr/local/java/jdk1.4/jre/lib/jsse.jar
          :/usr/local/java/jdk1.4/jre/lib/plugin.jar:/usr/local/java/jdk1.4/jre/lib/rt.jar:/usr/local/java/jdk1.4/jre/lib/sunrsasign.jar
          [javac] org/apache/derby/iapi/error/ExceptionSeverity.java added as org/apache/derby/iapi/error/ExceptionSeverity.class doesn't exist.
          [javac] org/apache/derby/iapi/error/PassThroughException.java added as org/apache/derby/iapi/error/PassThroughException.class doesn't exist.
          [javac] org/apache/derby/iapi/error/PublicAPI.java added as org/apache/derby/iapi/error/PublicAPI.class doesn't exist.
          [javac] org/apache/derby/iapi/error/StandardException.java added as org/apache/derby/iapi/error/StandardException.class doesn't exist.
          [javac] Compiling 4 source files to /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/classes
          [javac] Using modern compiler
          [javac] Compilation arguments:
          [javac] '-nowarn'
          [javac] '-d'
          [javac] '/export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/classes'
          [javac] '-classpath'
          [javac] '/export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/classes:/usr/local/java/jdk1.4/jre/lib/charsets.jar
          :/usr/local/java/jdk1.4/jre/lib/jce.jar:/usr/local/java/jdk1.4/jre/lib/jsse.jar:/usr/local/java/jdk1.4/jre/lib/plugin.jar
          :/usr/local/java/jdk1.4/jre/lib/rt.jar:/usr/local/java/jdk1.4/jre/lib/sunrsasign.jar'
          [javac] '-sourcepath'
          [javac] '/export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine'
          [javac] '-target'
          [javac] '1.4'
          [javac] '-bootclasspath'
          [javac] '/export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/tools/java/empty.jar'
          [javac] '-g'
          [javac] '-source'
          [javac] '1.4'
          [javac]
          [javac] The ' characters around the executable and arguments are
          [javac] not part of the command.
          [javac] Files to be compiled:
          [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/error/ExceptionSeverity.java
          [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/error/PassThroughException.java
          [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/error/PublicAPI.java
          [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/error/StandardException.java
          [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:20: package javax.management does not exist
          [javac] import javax.management.MBeanServer;
          [javac] ^
          [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:21: package javax.management does not exist
          [javac] import javax.management.ObjectName;
          [javac] ^
          etc..

          (Edit: splitted up the classpath to make it more readable)

          Show
          John H. Embretsen added a comment - - edited I am trying to get an updated version of the latest patch to compile, but I'm having the same problem Rick reported on Nov 20. After putting some echo statements into some of the build.xml files and running with verbose and debug flags, I still don't know why it happens. I have added jmxri.jar to tools/java as Bernt mentioned, but it doesn't help (it doesn't seem like it's being picked up into the compile classpath anyway). In any case, I have no idea why ant is reporting errors for org/apache/derby/iapi/services/mbeans/ManagementService when it is supposed to compile files in org/apache/derby/iapi/error/ only, as part of the compile_iapi_error_jsr169 target in java/engine/org/apache/derby/iapi/reference/build.xml. From what I can tell, the patch does not touch any files in org/apache/derby/iapi/error/. Any ideas? (...) [echo] Done with compile_reference [ant] Exiting /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/reference/build.xml. [echo] First javac in compile_iapi_error_jsr169 [echo] compile.classpath is: /usr/local/java/jdk1.4/jre/lib/charsets.jar:/usr/local/java/jdk1.4/jre/lib/jce.jar:/usr/local/java/jdk1.4/jre/lib/jsse.jar :/usr/local/java/jdk1.4/jre/lib/plugin.jar:/usr/local/java/jdk1.4/jre/lib/rt.jar:/usr/local/java/jdk1.4/jre/lib/sunrsasign.jar [javac] org/apache/derby/iapi/error/ExceptionSeverity.java added as org/apache/derby/iapi/error/ExceptionSeverity.class doesn't exist. [javac] org/apache/derby/iapi/error/PassThroughException.java added as org/apache/derby/iapi/error/PassThroughException.class doesn't exist. [javac] org/apache/derby/iapi/error/PublicAPI.java added as org/apache/derby/iapi/error/PublicAPI.class doesn't exist. [javac] org/apache/derby/iapi/error/StandardException.java added as org/apache/derby/iapi/error/StandardException.class doesn't exist. [javac] Compiling 4 source files to /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/classes [javac] Using modern compiler [javac] Compilation arguments: [javac] '-nowarn' [javac] '-d' [javac] '/export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/classes' [javac] '-classpath' [javac] '/export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/classes:/usr/local/java/jdk1.4/jre/lib/charsets.jar :/usr/local/java/jdk1.4/jre/lib/jce.jar:/usr/local/java/jdk1.4/jre/lib/jsse.jar:/usr/local/java/jdk1.4/jre/lib/plugin.jar :/usr/local/java/jdk1.4/jre/lib/rt.jar:/usr/local/java/jdk1.4/jre/lib/sunrsasign.jar' [javac] '-sourcepath' [javac] '/export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine' [javac] '-target' [javac] '1.4' [javac] '-bootclasspath' [javac] '/export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/tools/java/empty.jar' [javac] '-g' [javac] '-source' [javac] '1.4' [javac] [javac] The ' characters around the executable and arguments are [javac] not part of the command. [javac] Files to be compiled: [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/error/ExceptionSeverity.java [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/error/PassThroughException.java [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/error/PublicAPI.java [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/error/StandardException.java [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:20: package javax.management does not exist [javac] import javax.management.MBeanServer; [javac] ^ [javac] /export/home/tmp/je159969/Derby/sandboxes/codeTrunk/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:21: package javax.management does not exist [javac] import javax.management.ObjectName; [javac] ^ etc.. (Edit: splitted up the classpath to make it more readable)
          Hide
          John H. Embretsen added a comment -

          Assigning this to myself since Bernt is not available to work on this at the moment. I will try to grab the latest patch and fill in the blanks required to get it to apply cleanly to trunk and compile. Then I intend to start looking at how we can test this feature.

          Show
          John H. Embretsen added a comment - Assigning this to myself since Bernt is not available to work on this at the moment. I will try to grab the latest patch and fill in the blanks required to get it to apply cleanly to trunk and compile. Then I intend to start looking at how we can test this feature.
          Hide
          Rick Hillegas added a comment -

          Thanks, Bernt. Here is a bit more feedback on the patch:

          Error applying patch:

          patching file java/drda/org/apache/derby/drda/server.policy
          Hunk #2 FAILED at 147.
          1 out of 2 hunks FAILED – saving rejects to file java/drda/org/apache/derby/drda/server.policy.rej

          +++ java/engine/org/apache/derby/impl/services/build.xml

          This looks like a typo: optimize="$

          {optimijavacze}

          "

          +++ java/drda/org/apache/derby/drda/server.policy

          You will want to add these new permissions to template.policy also.

          +++ java/engine/org/apache/derby/impl/services/mbeans/MDatabaseMBean.java

          Hard to evaluate because no implementation is provided.

          Show
          Rick Hillegas added a comment - Thanks, Bernt. Here is a bit more feedback on the patch: Error applying patch: patching file java/drda/org/apache/derby/drda/server.policy Hunk #2 FAILED at 147. 1 out of 2 hunks FAILED – saving rejects to file java/drda/org/apache/derby/drda/server.policy.rej +++ java/engine/org/apache/derby/impl/services/build.xml This looks like a typo: optimize="$ {optimijavacze} " +++ java/drda/org/apache/derby/drda/server.policy You will want to add these new permissions to template.policy also. +++ java/engine/org/apache/derby/impl/services/mbeans/MDatabaseMBean.java Hard to evaluate because no implementation is provided.
          Hide
          Bernt M. Johnsen added a comment -

          Thanks Rick. JMX was not part of the 1.4 SDK. For the time being, you
          need to download JMX 1.2.1 RI and place it in tools/java (start your
          browing at
          http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/)

          I'll look into the matter.

          Show
          Bernt M. Johnsen added a comment - Thanks Rick. JMX was not part of the 1.4 SDK. For the time being, you need to download JMX 1.2.1 RI and place it in tools/java (start your browing at http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/ ) I'll look into the matter.
          Hide
          Rick Hillegas added a comment -

          Thanks for the patch, Bernt. Here's some quick initial feedback: When I apply this patch to a clean subversion client, I get build errors when I run "ant all". It seems that the compilation of the JMX support is being pulled in by the compile_references target. That target uses the jdk1.4 libraries and so doesn't see the javax.management package:

          compile_reference:
          [javac] Compiling 4 source files to /Users/rh161140/derby/mainline/trunk/classes
          [javac] /Users/rh161140/derby/mainline/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:20: package javax.management does not exist
          [javac] import javax.management.MBeanServer;
          [javac] ^
          [javac] /Users/rh161140/derby/mainline/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:21: package javax.management does not exist
          [javac] import javax.management.ObjectName;
          [javac] ^
          [javac] /Users/rh161140/derby/mainline/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:57: cannot find symbol
          [javac] symbol : class MBeanServer
          [javac] location: interface org.apache.derby.iapi.services.mbeans.ManagementService
          [javac] public MBeanServer getMBeanServer();
          [javac] ^
          [javac] /Users/rh161140/derby/mainline/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:93: cannot find symbol
          [javac] symbol : class ObjectName
          [javac] location: interface org.apache.derby.iapi.services.mbeans.ManagementService
          [javac] public void registerMBean(Object bean, ObjectName name) throws Exception;
          [javac] ^
          [javac] 4 errors

          BUILD FAILED

          Show
          Rick Hillegas added a comment - Thanks for the patch, Bernt. Here's some quick initial feedback: When I apply this patch to a clean subversion client, I get build errors when I run "ant all". It seems that the compilation of the JMX support is being pulled in by the compile_references target. That target uses the jdk1.4 libraries and so doesn't see the javax.management package: compile_reference: [javac] Compiling 4 source files to /Users/rh161140/derby/mainline/trunk/classes [javac] /Users/rh161140/derby/mainline/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:20: package javax.management does not exist [javac] import javax.management.MBeanServer; [javac] ^ [javac] /Users/rh161140/derby/mainline/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:21: package javax.management does not exist [javac] import javax.management.ObjectName; [javac] ^ [javac] /Users/rh161140/derby/mainline/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:57: cannot find symbol [javac] symbol : class MBeanServer [javac] location: interface org.apache.derby.iapi.services.mbeans.ManagementService [javac] public MBeanServer getMBeanServer(); [javac] ^ [javac] /Users/rh161140/derby/mainline/trunk/java/engine/org/apache/derby/iapi/services/mbeans/ManagementService.java:93: cannot find symbol [javac] symbol : class ObjectName [javac] location: interface org.apache.derby.iapi.services.mbeans.ManagementService [javac] public void registerMBean(Object bean, ObjectName name) throws Exception; [javac] ^ [javac] 4 errors BUILD FAILED
          Hide
          Bernt M. Johnsen added a comment -

          ... and now with license granted...

          Show
          Bernt M. Johnsen added a comment - ... and now with license granted...
          Hide
          Bernt M. Johnsen added a comment -

          New version (second last?). Authentication implemented. Propagation of SQLExceptions to JMX server. Some new functions (bootDB, autnehticateAs etc).

          Show
          Bernt M. Johnsen added a comment - New version (second last?). Authentication implemented. Propagation of SQLExceptions to JMX server. Some new functions (bootDB, autnehticateAs etc).
          Hide
          Bernt M. Johnsen added a comment -

          New patch pretty soon.

          Show
          Bernt M. Johnsen added a comment - New patch pretty soon.
          Hide
          Dyre Tjeldvoll added a comment -

          Hi Bernt and Rick,

          What is the status of this issue? Will there be another rev of the patch? If so; I think it would be good to uncheck 'Patch available'.

          Show
          Dyre Tjeldvoll added a comment - Hi Bernt and Rick, What is the status of this issue? Will there be another rev of the patch? If so; I think it would be good to uncheck 'Patch available'.
          Hide
          Bernt M. Johnsen added a comment -

          Rick: Thanks for your comments.

          WRT: exceptions. From my point of view, it would be best to just
          propagate SQLExceptions through JMX back to the client. That would
          give a Derby-specific management application the best possibilites of
          a meaningful user interface. The problem is the generic JMX-consoles,
          like JConsole, which then would get an UnmarshalException or a
          ClassNotFoundException unless the user points to the derby Jar-files.

          With JConsole, this is somewhat awkward, since the user has to start
          JConsole e.g. like this:

          jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/tools.jar:db-derby-10.3.1.4-lib/lib/derby.jar

          Anyway, I guess any compliant generic JMX client would have this
          capability one way or another since the JMX specification, as I read
          it, allows MBean methods to throw any exception.

          Show
          Bernt M. Johnsen added a comment - Rick: Thanks for your comments. WRT: exceptions. From my point of view, it would be best to just propagate SQLExceptions through JMX back to the client. That would give a Derby-specific management application the best possibilites of a meaningful user interface. The problem is the generic JMX-consoles, like JConsole, which then would get an UnmarshalException or a ClassNotFoundException unless the user points to the derby Jar-files. With JConsole, this is somewhat awkward, since the user has to start JConsole e.g. like this: jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/tools.jar:db-derby-10.3.1.4-lib/lib/derby.jar Anyway, I guess any compliant generic JMX client would have this capability one way or another since the JMX specification, as I read it, allows MBean methods to throw any exception.
          Hide
          Rick Hillegas added a comment -

          Thanks for rev 7 of the patch, Bernt. A couple more comments:

          1) MDatabase.checkTableConsistency() - If this method is going to
          return a String rather than a boolean, then I wonder if the failure
          case should just return the message from the exception--rather than
          repackaging that message as a SQLException.

          2) MDatabase.getProperty() - This method seems to map states
          to strings as follows:

          SQLException -> silently swallowed and replaced with a default value
          Property not set -> default value
          Property is set -> its value
          Not connected to database -> the string "N/A"

          I think that mapping the first two states to the same outcome may turn
          out to be confusing.

          3) MDatabase.getStorageTempDirectory() - What is the default returned
          by this method? Is STORAGE_TEMP_DIRECTORY always supposed to be set?
          If not, this method seems to return a misleading default.

          4) DerbySystemMBean.setStoragePageSize() - I think that the conversion
          of 4k, 8k, 16k, and 32k to stringified integers should happen before the initial
          check for legal values. As it stands, I don't think the conversion
          will ever be executed.

          Show
          Rick Hillegas added a comment - Thanks for rev 7 of the patch, Bernt. A couple more comments: 1) MDatabase.checkTableConsistency() - If this method is going to return a String rather than a boolean, then I wonder if the failure case should just return the message from the exception--rather than repackaging that message as a SQLException. 2) MDatabase.getProperty() - This method seems to map states to strings as follows: SQLException -> silently swallowed and replaced with a default value Property not set -> default value Property is set -> its value Not connected to database -> the string "N/A" I think that mapping the first two states to the same outcome may turn out to be confusing. 3) MDatabase.getStorageTempDirectory() - What is the default returned by this method? Is STORAGE_TEMP_DIRECTORY always supposed to be set? If not, this method seems to return a misleading default. 4) DerbySystemMBean.setStoragePageSize() - I think that the conversion of 4k, 8k, 16k, and 32k to stringified integers should happen before the initial check for legal values. As it stands, I don't think the conversion will ever be executed.
          Hide
          Bernt M. Johnsen added a comment -

          New rev. 7 of the patch. Hardcoded property names removed and default values filled in and some more tidying of code. The calls from NetworkServerControlMgt has do be through NetworkServerControlImpl since the MBean is registered from there and there is no backpointer to NetworkServerControl.

          Show
          Bernt M. Johnsen added a comment - New rev. 7 of the patch. Hardcoded property names removed and default values filled in and some more tidying of code. The calls from NetworkServerControlMgt has do be through NetworkServerControlImpl since the MBean is registered from there and there is no backpointer to NetworkServerControl.
          Hide
          Bernt M. Johnsen added a comment -

          Thanks Rick. I agree to your comments on the getter/setter methods. Internationalization should also be done and hard-coded property names will be replaced with constants from Property.java.

          WRT user managemement: This has to evolve into a better interface, and you should consider what now there as a proof of concept.

          There is no calls to NetworkServerControlImpl that can't be done through NetworkServerControl, so I'm not sure that there is a reason. I'll look into it.

          Show
          Bernt M. Johnsen added a comment - Thanks Rick. I agree to your comments on the getter/setter methods. Internationalization should also be done and hard-coded property names will be replaced with constants from Property.java. WRT user managemement: This has to evolve into a better interface, and you should consider what now there as a proof of concept. There is no calls to NetworkServerControlImpl that can't be done through NetworkServerControl, so I'm not sure that there is a reason. I'll look into it.
          Hide
          Rick Hillegas added a comment -

          Thanks for the patch, Bernt. I have a couple comments:

          General:

          Bean method names: Maybe the accessor method names could include all
          of the legs of the property names (except for the leading "derby").
          So, for instance, getMinimumRecordSize() would become
          getStorageMinimumRecordSize(). Then getMonitor() would become
          getLocksMonitor() and make a little more sense. This provides a
          regular pattern for others to follow when more properties are added
          and I think, in the long run, will result in less confusion and fewer
          name collisions.

          What are our internationalization expectations here? I see that
          exceptions have hard-coded English strings in them.

          Couldn't we use the constants in Property.java rather than hard-coded
          property strings? If the problem is that Property.java doesn't live in
          the correct jar file, then this can be solved the way we tackled this
          problem for SQLState.java.

          MDatabase:

          I think that user management method names should indicate that we are
          configuring users for the BUILTIN mode. Could we have a method for
          deleting a BUILTIN user too? I think that the user management methods
          should raise an exception if BUILTIN authentication hasn't been
          enabled--I think it would be confusing to be able to call these
          methods if they aren't really going to have an effect.

          Does anything prevent more than one client from manipulating a
          database at the same time? Should we build some sort of lock to
          control this?

          NetworkServerControlMgmt

          The header comment says that this class uses callbacks to
          NetworkServerControlImpl rather than calling
          NetworkServerControl. That makes me wonder, why?

          Show
          Rick Hillegas added a comment - Thanks for the patch, Bernt. I have a couple comments: General: Bean method names: Maybe the accessor method names could include all of the legs of the property names (except for the leading "derby"). So, for instance, getMinimumRecordSize() would become getStorageMinimumRecordSize(). Then getMonitor() would become getLocksMonitor() and make a little more sense. This provides a regular pattern for others to follow when more properties are added and I think, in the long run, will result in less confusion and fewer name collisions. What are our internationalization expectations here? I see that exceptions have hard-coded English strings in them. Couldn't we use the constants in Property.java rather than hard-coded property strings? If the problem is that Property.java doesn't live in the correct jar file, then this can be solved the way we tackled this problem for SQLState.java. MDatabase: I think that user management method names should indicate that we are configuring users for the BUILTIN mode. Could we have a method for deleting a BUILTIN user too? I think that the user management methods should raise an exception if BUILTIN authentication hasn't been enabled--I think it would be confusing to be able to call these methods if they aren't really going to have an effect. Does anything prevent more than one client from manipulating a database at the same time? Should we build some sort of lock to control this? NetworkServerControlMgmt The header comment says that this class uses callbacks to NetworkServerControlImpl rather than calling NetworkServerControl. That makes me wonder, why?
          Hide
          Rick Hillegas added a comment -

          This network server control bean implemented by this JIRA adds another way to shutdown a server. This needs to coordinate with the shutdown improvements being introduced by DERBY-2109.

          Show
          Rick Hillegas added a comment - This network server control bean implemented by this JIRA adds another way to shutdown a server. This needs to coordinate with the shutdown improvements being introduced by DERBY-2109 .
          Hide
          Bernt M. Johnsen added a comment -

          New patch DERBY-1387-6. Fixed some problems with db stop in embedded mode. derby.jmx property renamed to derby.system.jmx

          Show
          Bernt M. Johnsen added a comment - New patch DERBY-1387 -6. Fixed some problems with db stop in embedded mode. derby.jmx property renamed to derby.system.jmx
          Hide
          Bernt M. Johnsen added a comment -

          New version. Works now with jar-files (pervious tests was run from classes) and with security manager. Also did some cleanup and improvements in the code.

          Show
          Bernt M. Johnsen added a comment - New version. Works now with jar-files (pervious tests was run from classes) and with security manager. Also did some cleanup and improvements in the code.
          Hide
          Bernt M. Johnsen added a comment -

          Rick> I'm not sure I understand how this would work. Does this mean that administration would only be possible on databases which have already been brought up by normal application activity?

          Yes, in the current patch, that is the case.

          Rick> What if I want to administer a database which has been taken down and has no connected users?

          We will also need a knob for enabling of mangement of databases that have not been booted or have been shut down.

          Rick> Maybe we should only allow JMX management of Derby if authentication is turned on. If we did that, would we need an extra JMX-enabling knob?

          Interesting thought. But I hope to avoid the following scenario (at least in the normal use-cases): Given that JMX authentication is turned on, the user has to 1) Authenticate to be allowed access to the JMX ssystem, 2) Authenticate to be allowed to manage the Derby system and 3) Authenticate again to be recognized as DB owner of a given database. In other words: Authenticate 3 times before being allowed to de something with a specific database through JMX.

          Show
          Bernt M. Johnsen added a comment - Rick> I'm not sure I understand how this would work. Does this mean that administration would only be possible on databases which have already been brought up by normal application activity? Yes, in the current patch, that is the case. Rick> What if I want to administer a database which has been taken down and has no connected users? We will also need a knob for enabling of mangement of databases that have not been booted or have been shut down. Rick> Maybe we should only allow JMX management of Derby if authentication is turned on. If we did that, would we need an extra JMX-enabling knob? Interesting thought. But I hope to avoid the following scenario (at least in the normal use-cases): Given that JMX authentication is turned on, the user has to 1) Authenticate to be allowed access to the JMX ssystem, 2) Authenticate to be allowed to manage the Derby system and 3) Authenticate again to be recognized as DB owner of a given database. In other words: Authenticate 3 times before being allowed to de something with a specific database through JMX.
          Hide
          Rick Hillegas added a comment -

          Bernt> In oder to be able to make the JMX-components comply with the Derby Authentication/Athorization scheme: Implement and BootedDatabasesMBean which will hold a list of booted databases in the system.

          I'm not sure I understand how this would work. Does this mean that administration would only be possible on databases which have already been brought up by normal application activity? What if I want to administer a database which has been taken down and has no connected users?

          Bernt> On second thought. Making the Derby JMX automatically available in the MBean server will make it impossible for the user to make some application with an embedded Derby db manageable thorugh JMX without also making Derby manageable thorugh JMX. I would think that this "all or nothing" granularity could be a problem for some applications.

          Maybe we should only allow JMX management of Derby if authentication is turned on. If we did that, would we need an extra JMX-enabling knob?

          Show
          Rick Hillegas added a comment - Bernt> In oder to be able to make the JMX-components comply with the Derby Authentication/Athorization scheme: Implement and BootedDatabasesMBean which will hold a list of booted databases in the system. I'm not sure I understand how this would work. Does this mean that administration would only be possible on databases which have already been brought up by normal application activity? What if I want to administer a database which has been taken down and has no connected users? Bernt> On second thought. Making the Derby JMX automatically available in the MBean server will make it impossible for the user to make some application with an embedded Derby db manageable thorugh JMX without also making Derby manageable thorugh JMX. I would think that this "all or nothing" granularity could be a problem for some applications. Maybe we should only allow JMX management of Derby if authentication is turned on. If we did that, would we need an extra JMX-enabling knob?
          Hide
          Bernt M. Johnsen added a comment -

          On second thought. Making the Derby JMX automatically available in the MBean server will make it impossible for the user to make some application with an embedded Derby db manageable thorugh JMX without also making Derby manageable thorugh JMX. I would think that this "all or nothing" granularity could be a problem for some applications. So we would need an explicit derby.system.jmx property for enabling the management service anyway. Any comments?

          Show
          Bernt M. Johnsen added a comment - On second thought. Making the Derby JMX automatically available in the MBean server will make it impossible for the user to make some application with an embedded Derby db manageable thorugh JMX without also making Derby manageable thorugh JMX. I would think that this "all or nothing" granularity could be a problem for some applications. So we would need an explicit derby.system.jmx property for enabling the management service anyway. Any comments?
          Hide
          Bernt M. Johnsen added a comment -

          Usability issue:

          Instead of controlling whether JMX for Derby is enabled or not by derby properties and/or command line options, this should be dependent on the availability of an MBean server. That is, JMX MBeans for Derby will be generated and registered if ManagerFactory.getPlatformMBeanServer() returns a non-null value. The Mbeans will not be available outside the VM unless the MBean server is instructed to do so. For Sun's VM, this is done by setting a system property, e.g. like -Dcom.sun.management.jmxremote

          When this is done, an JMX compliant application (e.g. JConsole) may connect to the MBean server if run on the same machine and from the same OS user as the VM is running. More properties may be set, to enable connections from another machine, enable/disable ssl etc. See e.g. http://java.sun.com/docs/books/tutorial/jmx/remote/jconsole.html

          Show
          Bernt M. Johnsen added a comment - Usability issue: Instead of controlling whether JMX for Derby is enabled or not by derby properties and/or command line options, this should be dependent on the availability of an MBean server. That is, JMX MBeans for Derby will be generated and registered if ManagerFactory.getPlatformMBeanServer() returns a non-null value. The Mbeans will not be available outside the VM unless the MBean server is instructed to do so. For Sun's VM, this is done by setting a system property, e.g. like -Dcom.sun.management.jmxremote When this is done, an JMX compliant application (e.g. JConsole) may connect to the MBean server if run on the same machine and from the same OS user as the VM is running. More properties may be set, to enable connections from another machine, enable/disable ssl etc. See e.g. http://java.sun.com/docs/books/tutorial/jmx/remote/jconsole.html
          Hide
          Bernt M. Johnsen added a comment -

          In oder to be able to make the JMX-components comply with the Derby Authentication/Athorization scheme: Implement and BootedDatabasesMBean which will hold a list of booted databases in the system. To manage a database, the user must provided the needed credentials for the DB Owner before the DatabaseMBean for the given database becomes available in the MBean server. This should address Rick's 3c) comment.

          Show
          Bernt M. Johnsen added a comment - In oder to be able to make the JMX-components comply with the Derby Authentication/Athorization scheme: Implement and BootedDatabasesMBean which will hold a list of booted databases in the system. To manage a database, the user must provided the needed credentials for the DB Owner before the DatabaseMBean for the given database becomes available in the MBean server. This should address Rick's 3c) comment.
          Hide
          Bernt M. Johnsen added a comment -

          Uploaded new patch DERBY-1387-4.diff (and DERBY-1387-4). I have corrected some JDBC driver use which did nok work in embedded mode. I have changed the exception handling to get rid of the System.out.println and ex.printStackTrace calls. Now the JMX classes dump the stack trace on derby.log and returns an exception back to the MBean server. It will is the JMX client's resposibility to report properly to the user and take the needed action (and possibly advise the user to look in derby.log). This patch is still not suitable for commit, but feel free to test it out e.g. with the jconsole.

          The authorization issues raised by Rick & Dan (thanks for your comments!) will be discussed and handeled at a later stage.

          Show
          Bernt M. Johnsen added a comment - Uploaded new patch DERBY-1387 -4.diff (and DERBY-1387 -4). I have corrected some JDBC driver use which did nok work in embedded mode. I have changed the exception handling to get rid of the System.out.println and ex.printStackTrace calls. Now the JMX classes dump the stack trace on derby.log and returns an exception back to the MBean server. It will is the JMX client's resposibility to report properly to the user and take the needed action (and possibly advise the user to look in derby.log). This patch is still not suitable for commit, but feel free to test it out e.g. with the jconsole. The authorization issues raised by Rick & Dan (thanks for your comments!) will be discussed and handeled at a later stage.
          Hide
          Daniel John Debrunner added a comment -

          Rick> So if the DBA uses system procedures to read the passwords, hashed values come back.

          I don't think so. I think NULL will be returned for a password lookup using the get database property method.

          Only is the DBA has read acces to the raw files, or uses the security hole in export could they see the hash values of the passwords.

          Show
          Daniel John Debrunner added a comment - Rick> So if the DBA uses system procedures to read the passwords, hashed values come back. I don't think so. I think NULL will be returned for a password lookup using the get database property method. Only is the DBA has read acces to the raw files, or uses the security hole in export could they see the hash values of the passwords.
          Hide
          Rick Hillegas added a comment -

          I see. So if the DBA uses system procedures to read the passwords, hashed values come back.

          Starting another thread on this topic: Password management is generally handled by end-users. An administrator sets up an initial password and the end-user then changes it immediately afterward. I don't think our BUILTIN scheme works this way--although we could improve it by giving end-users a tool to change their passwords. The JMX-based password setting could be useful if we just see it as a way for the DBA to initialize passwords or forcibly reset them when the user forgets her credentials.

          Show
          Rick Hillegas added a comment - I see. So if the DBA uses system procedures to read the passwords, hashed values come back. Starting another thread on this topic: Password management is generally handled by end-users. An administrator sets up an initial password and the end-user then changes it immediately afterward. I don't think our BUILTIN scheme works this way--although we could improve it by giving end-users a tool to change their passwords. The JMX-based password setting could be useful if we just see it as a way for the DBA to initialize passwords or forcibly reset them when the user forgets her credentials.
          Hide
          Daniel John Debrunner added a comment -

          Rick> Nevertheless, even when stored in the database, aren't the plaintext passwords are still visible to the DBA?

          No. There are stored as hashed values of the password, similar to /etc/password in unix systems.

          Show
          Daniel John Debrunner added a comment - Rick> Nevertheless, even when stored in the database, aren't the plaintext passwords are still visible to the DBA? No. There are stored as hashed values of the password, similar to /etc/password in unix systems.
          Hide
          Rick Hillegas added a comment -

          Thanks for the clarification, Dan. Nevertheless, even when stored in the database, aren't the plaintext passwords are still visible to the DBA?

          Show
          Rick Hillegas added a comment - Thanks for the clarification, Dan. Nevertheless, even when stored in the database, aren't the plaintext passwords are still visible to the DBA?
          Hide
          Daniel John Debrunner added a comment -

          Rick> I do not think that we should be encouraging customers to use the BUILTIN authentication scheme. In that scheme passwords are stored in plaintext.

          Just to be clear, passwords are only stored in clear text when set at the system level (in derby.properties or as system properties). When stored in the database they are not stored in clear text.

          Show
          Daniel John Debrunner added a comment - Rick> I do not think that we should be encouraging customers to use the BUILTIN authentication scheme. In that scheme passwords are stored in plaintext. Just to be clear, passwords are only stored in clear text when set at the system level (in derby.properties or as system properties). When stored in the database they are not stored in clear text.
          Hide
          Rick Hillegas added a comment -

          Thanks to Ole Gunnar for the great functional spec. I have a couple comments:

          1) Under DatabaseMBean, the property associated with DatabaseName looks wrong to me.

          2) Under DatabaseMBean: I have reservations about the addDBUser() operation. I do not think that we should be encouraging customers to use the BUILTIN authentication scheme. In that scheme passwords are stored in plaintext. That seems very insecure to me. I think it's ok for testing purposes but not for a production environment. I recommend against exposing this operation.

          3) In general, I think we need to beef up the authorization story for this JMX-based administration before we expose it to customers. Incremental development is great, but I think real production usage requires more controls. If I understand the spec correctly, it seems that godlike administrative powers over all databases are granted to the VM's administrator. I think this is inappropriate for VMs which host other applications besides Derby. I propose the following:

          a) The VersionMBean looks pretty harmless to me. I don't think it needs more controls.

          b) I think that in order to get your hands on a SystemMBean or a NSCMBean, you should be forced to authenticate at the Derby system-wide level. Furthermore, this authentication should result in your being a system-wide DatabasePrincipal to whom the policy file grants 'permission org.apache.derby.security.SystemPermission "systemAdministration"'. For more information on this permissions scheme, see the functional spec for DERBY-2109.

          c) You must authenticate as the database's DBA in order to get your hands on the corresponding DatabaseMBean.

          4) Continuing on the topic of authorization: If I understand the spec correctly, it seems that, potentially, the Derby System Administrator and all of the Derby DBAs will be given the password for VM-wide JMX-based administration. In theory, this gives these users the ability to manipulate other applications running in the VM. The user guides should state clearly that these other applications are responsible for raising additional authorization hurdles if they are uncomfortable with these godlike powers that are granted to Derby super-users.

          Show
          Rick Hillegas added a comment - Thanks to Ole Gunnar for the great functional spec. I have a couple comments: 1) Under DatabaseMBean, the property associated with DatabaseName looks wrong to me. 2) Under DatabaseMBean: I have reservations about the addDBUser() operation. I do not think that we should be encouraging customers to use the BUILTIN authentication scheme. In that scheme passwords are stored in plaintext. That seems very insecure to me. I think it's ok for testing purposes but not for a production environment. I recommend against exposing this operation. 3) In general, I think we need to beef up the authorization story for this JMX-based administration before we expose it to customers. Incremental development is great, but I think real production usage requires more controls. If I understand the spec correctly, it seems that godlike administrative powers over all databases are granted to the VM's administrator. I think this is inappropriate for VMs which host other applications besides Derby. I propose the following: a) The VersionMBean looks pretty harmless to me. I don't think it needs more controls. b) I think that in order to get your hands on a SystemMBean or a NSCMBean, you should be forced to authenticate at the Derby system-wide level. Furthermore, this authentication should result in your being a system-wide DatabasePrincipal to whom the policy file grants 'permission org.apache.derby.security.SystemPermission "systemAdministration"'. For more information on this permissions scheme, see the functional spec for DERBY-2109 . c) You must authenticate as the database's DBA in order to get your hands on the corresponding DatabaseMBean. 4) Continuing on the topic of authorization: If I understand the spec correctly, it seems that, potentially, the Derby System Administrator and all of the Derby DBAs will be given the password for VM-wide JMX-based administration. In theory, this gives these users the ability to manipulate other applications running in the VM. The user guides should state clearly that these other applications are responsible for raising additional authorization hurdles if they are uncomfortable with these godlike powers that are granted to Derby super-users.
          Hide
          Bernt M. Johnsen added a comment -

          Great work, Ole Gunnar. I will take it from here.
          The following issues needs to be resolved:
          1) make compilation of the jmx parts optional and dependent on wether jmxri.jar is present in tools/java (jmxri.jar is necessary for compiling and for running jmx in jdk 1.4)
          2) make JMX work in embedded mode (seems to be a trivial issue)
          3) make the start command more simple for the default case
          4) testing
          5) Consider reducing the number of properties exposed through JMX. There is a conflict here between ease of use for the average user and the usability for the advanced user.

          Show
          Bernt M. Johnsen added a comment - Great work, Ole Gunnar. I will take it from here. The following issues needs to be resolved: 1) make compilation of the jmx parts optional and dependent on wether jmxri.jar is present in tools/java (jmxri.jar is necessary for compiling and for running jmx in jdk 1.4) 2) make JMX work in embedded mode (seems to be a trivial issue) 3) make the start command more simple for the default case 4) testing 5) Consider reducing the number of properties exposed through JMX. There is a conflict here between ease of use for the average user and the usability for the advanced user.
          Hide
          Ole Gunnar Borstad added a comment -

          DERBY-1387-3.diff

          Attaching a new version of the patch fixing some issues reported by Bernt (thank you for the help).

          The NSCMBean (networkservercontrol mbean) is moved to DRDA to fix a packaging issue.

          I have also edited compilepath.properties to add jmxri.jar. This is the JMX extension which should be a part of JDK1.5 and up. I was not able to build without adding the jar file explicitly though, probably because I have not set up the build files correctly. So to compile you need to download this jar file for now.

          Show
          Ole Gunnar Borstad added a comment - DERBY-1387 -3.diff Attaching a new version of the patch fixing some issues reported by Bernt (thank you for the help). The NSCMBean (networkservercontrol mbean) is moved to DRDA to fix a packaging issue. I have also edited compilepath.properties to add jmxri.jar. This is the JMX extension which should be a part of JDK1.5 and up. I was not able to build without adding the jar file explicitly though, probably because I have not set up the build files correctly. So to compile you need to download this jar file for now.
          Hide
          Ole Gunnar Borstad added a comment -

          DERBY-1387-2.diff
          DERBY-1387-2.stat

          Attaching a new patch fixing the compile issue Bernt reported.

          Show
          Ole Gunnar Borstad added a comment - DERBY-1387 -2.diff DERBY-1387 -2.stat Attaching a new patch fixing the compile issue Bernt reported.
          Hide
          Bernt M. Johnsen added a comment -

          The patch does not compile in my environment. Seems like something is missing in java/engine/org/apache/derby/iapi/build.xml which tries to compile the JMX stuff with JDK 1.4.

          Show
          Bernt M. Johnsen added a comment - The patch does not compile in my environment. Seems like something is missing in java/engine/org/apache/derby/iapi/build.xml which tries to compile the JMX stuff with JDK 1.4.
          Hide
          Ole Gunnar Borstad added a comment -

          Attaching a functional specification for this feature.

          Show
          Ole Gunnar Borstad added a comment - Attaching a functional specification for this feature.
          Hide
          Ole Gunnar Borstad added a comment -

          DERBY-1387-1.diff
          DERBY-1387-1.stat

          This patch adds a management and monitoring service to Derby through JMX extentions. The service is designed for a client/server environment. The patch is a refactor and expansion of Sanket Sharma's work.

          No tests have been run. Only functional testing of the features has been done. No test cases are included.

          • FEATURES

          1) Deployment
          Derby can be monitored locally or remotely. Both configurations use the JVM's built-in instrumentation as the JMX agent, where password authentication and SSL support is provided.

          The management functionality is implemented through 4 managed beans (MBeans):

          2) VersionMBean
          View Derby version information.
          No operations are provided.

          3) DerbySystemMBean
          View and change system properties. Changes to properties are not persisted in derby.properties, because currently no such mechanism exists in the Derby engine. Default values are displayed as <default>, because they are not accessible through the java Properties. Derby properties are visible through Properties only if set explicitly. There has been discussion on derby-dev on this issue, but no JIRA-issue is up yet.

          4) MDatabaseMBean
          All booted databases are wrapped by a bean which provides access to database properties. Changes are persistent.
          The following operations are provided:
          Stopping the database, backup, checking table consistency and adding database users. There is also an operation to connect as a user. This authenticates the management service and is to be used if Derby authentication is enabled.

          5) NSCMBean
          The public API of NetworkServerControl is made available through this bean. All methods of this API can now be invoked from a remote machine through the management service. This includes ping, shutdown, logging etc.

          • HOW TO BUILD AND RUN

          Build files are edited to compile the management service. The ManagementService and BasicManagementService classes are compiled with JDK 1.5. My edits of the build files might not be optimal so I would appreciate if anyone can review those.

          Run

          Start Derby with management by using the system property "derby.jmx". The property value is not relevant. Certain other system properties must also be set when starting the server-JVM to enable the built-in JMX agent. These properties vary with local/remote and security configurations. For details refer to the following URL:

          http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html

          This is an example of running with JMX agent enabled for local management and without security features:
          java -Dcom.sun.management.jmxremote -Dderby.jmx=true org.apache.derby.drda.NetworkServerControl start -noSecurityManager

          JConsole (found in JDK_HOME/bin) can be used as the management GUI. Details in above URL.

          Implementation details

          • JVM platform instrumentation
            The JVM has built-in instrumentation (from JDK 1.5) that enables you to monitor and manage the JVM and instrumented applications using JMX. The Derby management service uses this for both local and remote management.
          • Modifications to Derby engine code
            Derby boots the management service when the system property 'derby.jmx' is set. The module is loaded and started through the Derby monitor as done for other services. The changes to the Derby code are small and should hopefully not be a problem. MBeans are created when instrumented Derby resources are booted. For example, BasicDatabase.boot() instantiates an instance of MDatabaseMBean.
          • ManagementService
            This is the management service interface (found in org.apache.derby.iapi.services.mbeans), which holds the MBeanServer and methods to create beans. All beans should be created through this interface. To expand with more beans, one needs to find the appropriate Derby code to instrument, code a bean for it, and make factory methods in the ManagementService.

          All beans are implemented as Standard MBeans.

          API documentation is provided with the source code. The attached svn stat file lists the modifications and added files.

          Show
          Ole Gunnar Borstad added a comment - DERBY-1387 -1.diff DERBY-1387 -1.stat This patch adds a management and monitoring service to Derby through JMX extentions. The service is designed for a client/server environment. The patch is a refactor and expansion of Sanket Sharma's work. No tests have been run. Only functional testing of the features has been done. No test cases are included. FEATURES 1) Deployment Derby can be monitored locally or remotely. Both configurations use the JVM's built-in instrumentation as the JMX agent, where password authentication and SSL support is provided. The management functionality is implemented through 4 managed beans (MBeans): 2) VersionMBean View Derby version information. No operations are provided. 3) DerbySystemMBean View and change system properties. Changes to properties are not persisted in derby.properties, because currently no such mechanism exists in the Derby engine. Default values are displayed as <default>, because they are not accessible through the java Properties. Derby properties are visible through Properties only if set explicitly. There has been discussion on derby-dev on this issue, but no JIRA-issue is up yet. 4) MDatabaseMBean All booted databases are wrapped by a bean which provides access to database properties. Changes are persistent. The following operations are provided: Stopping the database, backup, checking table consistency and adding database users. There is also an operation to connect as a user. This authenticates the management service and is to be used if Derby authentication is enabled. 5) NSCMBean The public API of NetworkServerControl is made available through this bean. All methods of this API can now be invoked from a remote machine through the management service. This includes ping, shutdown, logging etc. HOW TO BUILD AND RUN Build files are edited to compile the management service. The ManagementService and BasicManagementService classes are compiled with JDK 1.5. My edits of the build files might not be optimal so I would appreciate if anyone can review those. Run Start Derby with management by using the system property "derby.jmx". The property value is not relevant. Certain other system properties must also be set when starting the server-JVM to enable the built-in JMX agent. These properties vary with local/remote and security configurations. For details refer to the following URL: http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html This is an example of running with JMX agent enabled for local management and without security features: java -Dcom.sun.management.jmxremote -Dderby.jmx=true org.apache.derby.drda.NetworkServerControl start -noSecurityManager JConsole (found in JDK_HOME/bin) can be used as the management GUI. Details in above URL. Implementation details JVM platform instrumentation The JVM has built-in instrumentation (from JDK 1.5) that enables you to monitor and manage the JVM and instrumented applications using JMX. The Derby management service uses this for both local and remote management. Modifications to Derby engine code Derby boots the management service when the system property 'derby.jmx' is set. The module is loaded and started through the Derby monitor as done for other services. The changes to the Derby code are small and should hopefully not be a problem. MBeans are created when instrumented Derby resources are booted. For example, BasicDatabase.boot() instantiates an instance of MDatabaseMBean. ManagementService This is the management service interface (found in org.apache.derby.iapi.services.mbeans), which holds the MBeanServer and methods to create beans. All beans should be created through this interface. To expand with more beans, one needs to find the appropriate Derby code to instrument, code a bean for it, and make factory methods in the ManagementService. All beans are implemented as Standard MBeans. API documentation is provided with the source code. The attached svn stat file lists the modifications and added files.
          Hide
          Ole Gunnar Borstad added a comment -

          I want to read and write database-wide properties dynamically. Tuning Derby document says to use the system procedures through SQL statements. Does this mean I need to setup a connection? Is there any reason not to have a connection in a management service, i.e exposing Derby of additional overhead etc.?

          Show
          Ole Gunnar Borstad added a comment - I want to read and write database-wide properties dynamically. Tuning Derby document says to use the system procedures through SQL statements. Does this mean I need to setup a connection? Is there any reason not to have a connection in a management service, i.e exposing Derby of additional overhead etc.?
          Hide
          Ole Gunnar Borstad added a comment -

          This patch is a clean version of derbyjmx.patch which was corrupted by multiple entries for each source file. No additional features are added. For those who had trouble using the old patch, try this one.

          Please note that two JAR files are needed in the classpath to be able to build/run with this patch:

          1. Apache Commons Modeler- commons-modeler-2.0.1.jar
          http://jakarta.apache.org/site/downloads/downloads_commons-modeler.cgi

          2. Apache Commons Logging - commons-logging-1.1.jar
          http://jakarta.apache.org/site/downloads/downloads_commons-logging.cgi

          Show
          Ole Gunnar Borstad added a comment - This patch is a clean version of derbyjmx.patch which was corrupted by multiple entries for each source file. No additional features are added. For those who had trouble using the old patch, try this one. Please note that two JAR files are needed in the classpath to be able to build/run with this patch: 1. Apache Commons Modeler- commons-modeler-2.0.1.jar http://jakarta.apache.org/site/downloads/downloads_commons-modeler.cgi 2. Apache Commons Logging - commons-logging-1.1.jar http://jakarta.apache.org/site/downloads/downloads_commons-logging.cgi
          Hide
          David Van Couvering added a comment -

          No matter what I try I can't get past the error saying "javax.management not found." I have explicitly set my JAVA_HOME to the 1.6 JDK and the 1.5 JDK to no success. Is this because javax.management requires a jar file that is independent of the JDK, even when you're building with JDK 1.5 or 1.6?

          Show
          David Van Couvering added a comment - No matter what I try I can't get past the error saying "javax.management not found." I have explicitly set my JAVA_HOME to the 1.6 JDK and the 1.5 JDK to no success. Is this because javax.management requires a jar file that is independent of the JDK, even when you're building with JDK 1.5 or 1.6?
          Hide
          Andreas Korneliussen added a comment -

          Hi, thanks for providing this patch. I have tried compiling it, and I do have the following comments:

          1. It seems that most of the classes have multiple entries in the patch file, so when applying the patch, I got the same class multiple times in the same file.

          2. After fixing 1, all code, except one class seems to be able to compile with JDK 1.4. The class which depends on JDK 1.5 is BasicManagementService. It uses java.lang.management.ManagementFactory to create the MBeanServer. Instead I would propose simply using javax.management.MBeanServerFactory. This would allow the code to be compiled on JDK 1.4.

          If you do need the JDK 1.5 libraries, the build files must be set up so that it is still possible to compile on JDK 1.4 (ie. by skipping the JMX targets).

          Show
          Andreas Korneliussen added a comment - Hi, thanks for providing this patch. I have tried compiling it, and I do have the following comments: 1. It seems that most of the classes have multiple entries in the patch file, so when applying the patch, I got the same class multiple times in the same file. 2. After fixing 1, all code, except one class seems to be able to compile with JDK 1.4. The class which depends on JDK 1.5 is BasicManagementService. It uses java.lang.management.ManagementFactory to create the MBeanServer. Instead I would propose simply using javax.management.MBeanServerFactory. This would allow the code to be compiled on JDK 1.4. If you do need the JDK 1.5 libraries, the build files must be set up so that it is still possible to compile on JDK 1.4 (ie. by skipping the JMX targets).
          Hide
          Sanket Sharma added a comment -

          Submitting The Code for Derby's JMX Functionality as part of my Google SOC project.

          The patch addresses the following:

          1) Start derby with monitoring extensions enabled. To start derby with monitoring extensions, you need to set derby.jmx environment variable while starting the JVM

          2) Derby can be monitored locally or remotely. Password authentication and SSL support is provided via SUN's RMI connector implementations. See the following URL for details
          http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html

          3) The MBeans can be viewed in JConsole.

          4) This implementation supports three different MBeans:

          ->SystemMBean for viewing and changing system properties. Please note that as of now changes are not persisted to derby.properties file
          ->DatabaseMBean, one per database. Any changes to database properties are persisted.
          ->VersionMBean for viewing DerbyVersion Information.

          The code was built using JDK1.6 and uses Sun's PlatformMBean server for exposing system health and Derby's MBeans.

          The code is up for review and contains the following files:

          iapi/services/mbeans/ManagementService.java Interface declaration for management service
          /iapi/services/mbeans/MBeanFactory Factory interface for creating MBeans

          /impl/services/mbeans/BasicManagementService Management service that starts up MBeanServer and Registry
          /impl/services/mbeans/BasicMBeanFactory Implementation of MBeanFactory interface
          /impl/services/mbeans/DerbySystemMBean For exposing derby system properties
          /impl/services/mbeans/DatabaseMBean For exposing a database's properties and operation for Stopping a database
          /impl/services/mbeans/mbeans-descriptors.xml MBean descriptor used by Apache Commons modeler framework.

          tools/jars/mbeansDBMSclass.properties lists the classes, so that they may be included in the derby jar.

          Minor changes to build files and properties files have also been made to compile the code.

          5) The JMX functionality was built using the Apache modeler commons framework available at http://commons.apache.org

          6) This patch DOES NOT include any test cases. I will be posting them in a seperate patch, as the test cases themselves will amount to substantial code.

          7)API docs are provided with the files.

          I will really appreciate if someone can help me with identifying which properties are System properties, which are database properties and which can be modified at runtime. I referred to iapi.reference.Property.java. However, I felt descriptions for some properties were ambiguous. The next iteration will probably be a fix for such properties.

          I welcome any comments.

          Best Regards,
          Sanket Sharma

          Show
          Sanket Sharma added a comment - Submitting The Code for Derby's JMX Functionality as part of my Google SOC project. The patch addresses the following: 1) Start derby with monitoring extensions enabled. To start derby with monitoring extensions, you need to set derby.jmx environment variable while starting the JVM 2) Derby can be monitored locally or remotely. Password authentication and SSL support is provided via SUN's RMI connector implementations. See the following URL for details http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html 3) The MBeans can be viewed in JConsole. 4) This implementation supports three different MBeans: ->SystemMBean for viewing and changing system properties. Please note that as of now changes are not persisted to derby.properties file ->DatabaseMBean, one per database. Any changes to database properties are persisted. ->VersionMBean for viewing DerbyVersion Information. The code was built using JDK1.6 and uses Sun's PlatformMBean server for exposing system health and Derby's MBeans. The code is up for review and contains the following files: iapi/services/mbeans/ManagementService.java Interface declaration for management service /iapi/services/mbeans/MBeanFactory Factory interface for creating MBeans /impl/services/mbeans/BasicManagementService Management service that starts up MBeanServer and Registry /impl/services/mbeans/BasicMBeanFactory Implementation of MBeanFactory interface /impl/services/mbeans/DerbySystemMBean For exposing derby system properties /impl/services/mbeans/DatabaseMBean For exposing a database's properties and operation for Stopping a database /impl/services/mbeans/mbeans-descriptors.xml MBean descriptor used by Apache Commons modeler framework. tools/jars/mbeansDBMSclass.properties lists the classes, so that they may be included in the derby jar. Minor changes to build files and properties files have also been made to compile the code. 5) The JMX functionality was built using the Apache modeler commons framework available at http://commons.apache.org 6) This patch DOES NOT include any test cases. I will be posting them in a seperate patch, as the test cases themselves will amount to substantial code. 7)API docs are provided with the files. I will really appreciate if someone can help me with identifying which properties are System properties, which are database properties and which can be modified at runtime. I referred to iapi.reference.Property.java. However, I felt descriptions for some properties were ambiguous. The next iteration will probably be a fix for such properties. I welcome any comments. Best Regards, Sanket Sharma
          Hide
          Mike Matrigali added a comment -

          for lack of a better component made it services.

          Show
          Mike Matrigali added a comment - for lack of a better component made it services.
          Hide
          Sanket Sharma added a comment -

          David,

          Can you please tell me more about non functional requirements? " Things like performance (response time, thorughput, number of concurrent users), security, ease of use, etc " - how exactly can I put them in a requirements doc?

          Show
          Sanket Sharma added a comment - David, Can you please tell me more about non functional requirements? " Things like performance (response time, thorughput, number of concurrent users), security, ease of use, etc " - how exactly can I put them in a requirements doc?
          Hide
          Sanket Sharma added a comment -

          The requirement specification for monitoring and management extensions for Apache Derby has been drafted and put up on Derby Wiki for discussion. Developers and Users and requested to please look at the document and assign priorities to the features listed.

          You are requested to add a coloumn to the feature table by your name and put your version of priorities in the table.

          The wiki page maybe accessed at
          http://wiki.apache.org/db-derby/_Requirement_Specifications_for_Monitoring_%26_Management_Extensions_using_JMX
          Comments are welcome.

          Show
          Sanket Sharma added a comment - The requirement specification for monitoring and management extensions for Apache Derby has been drafted and put up on Derby Wiki for discussion. Developers and Users and requested to please look at the document and assign priorities to the features listed. You are requested to add a coloumn to the feature table by your name and put your version of priorities in the table. The wiki page maybe accessed at http://wiki.apache.org/db-derby/_Requirement_Specifications_for_Monitoring_%26_Management_Extensions_using_JMX Comments are welcome.
          Hide
          David Van Couvering added a comment -

          Hi, Sanket, thanks for a very nicely organized and formatted set of requirements!

          Here are my comments:

          • In Section 5.1 you are making the implicit assumption that the JMX functionality is targetted towards Derby being an enterprise-class database. I'm not sure if everyone would agree that's the sweet spot for Derby. If you look at the charter, the focus is on zero-administration, embeddability, standards-based, etc.

          That said, I think JMX is important (see "ease of use" and "standards-based"), but when looking at requirements I want to make sure we don't get too top-heavy...

          • You talk about two key areas that JMX can contribute to: monitoring and tuning. I would assume "tuning" includes general system configuration too. I can see all the existing Derby properties being exposed through a JMX interface.
          • Under proposed features, it would really help if each feature had its own unique identifier, like MON-1, MON-2, MAN-1, MAN-2, NOT-1, NOT-2, etc.
          • A well-thought-out and great list of features! There are a lot of features here. So, how to decide? Your separate table with priorities is one approach, but I'd like to propose something a little more interactive. I'd like the feature list section of the document to a Wiki page (or maybe it makes sense to migrate the entire document to Wik? - your call), and announce it on the derby-user and derby-dev lists. Each developers/user could add a column to the table and assign their priorities to each feature. This would be a chance for the community to get involved in picking which JMX features they'd like to see.

          If you do decide to stick with the current model, I highly recommend merging the priorities with the main feature table, so you are not constantly having to keep the two in synch, and so that readers don't have to keep switching back and forth between the two tables.

          • I'm a little confused about not providing remote access. Can you describe how an administrator who is not aJMX programmer can view, monitor, and tune an embedded Derby database that is not part of a larger application framework that provides JMX support? Whenever I talk to users, remote monitoring and administration are on the TOP of their list. I think this is required, but we do have to make sure we understand the security implications of such a topology.
          • A commonly missed aspect of requirements documents are non-functional requirements. These are "qualities" of the overall feature. Things like performance (response time, thorughput, number of concurrent users), security, ease of use, etc.
          • You had mentioned building our own JMX console. Is that still on the plate? Or do we just let existing JMX tools plug into our framework?
          Show
          David Van Couvering added a comment - Hi, Sanket, thanks for a very nicely organized and formatted set of requirements! Here are my comments: In Section 5.1 you are making the implicit assumption that the JMX functionality is targetted towards Derby being an enterprise-class database. I'm not sure if everyone would agree that's the sweet spot for Derby. If you look at the charter, the focus is on zero-administration, embeddability, standards-based, etc. That said, I think JMX is important (see "ease of use" and "standards-based"), but when looking at requirements I want to make sure we don't get too top-heavy... You talk about two key areas that JMX can contribute to: monitoring and tuning. I would assume "tuning" includes general system configuration too. I can see all the existing Derby properties being exposed through a JMX interface. Under proposed features, it would really help if each feature had its own unique identifier, like MON-1, MON-2, MAN-1, MAN-2, NOT-1, NOT-2, etc. A well-thought-out and great list of features! There are a lot of features here. So, how to decide? Your separate table with priorities is one approach, but I'd like to propose something a little more interactive. I'd like the feature list section of the document to a Wiki page (or maybe it makes sense to migrate the entire document to Wik? - your call), and announce it on the derby-user and derby-dev lists. Each developers/user could add a column to the table and assign their priorities to each feature. This would be a chance for the community to get involved in picking which JMX features they'd like to see. If you do decide to stick with the current model, I highly recommend merging the priorities with the main feature table, so you are not constantly having to keep the two in synch, and so that readers don't have to keep switching back and forth between the two tables. I'm a little confused about not providing remote access. Can you describe how an administrator who is not aJMX programmer can view, monitor, and tune an embedded Derby database that is not part of a larger application framework that provides JMX support? Whenever I talk to users, remote monitoring and administration are on the TOP of their list. I think this is required, but we do have to make sure we understand the security implications of such a topology. A commonly missed aspect of requirements documents are non-functional requirements. These are "qualities" of the overall feature. Things like performance (response time, thorughput, number of concurrent users), security, ease of use, etc. You had mentioned building our own JMX console. Is that still on the plate? Or do we just let existing JMX tools plug into our framework?
          Hide
          Sanket Sharma added a comment -

          Please download this file for JMX requirements. This is the updated version.

          Show
          Sanket Sharma added a comment - Please download this file for JMX requirements. This is the updated version.
          Hide
          Sanket Sharma added a comment -

          Requirements in a single html file.

          Show
          Sanket Sharma added a comment - Requirements in a single html file.
          Hide
          David Van Couvering added a comment -

          Hi, Sanket. Can you re-attach your documents as individual non-compressed files? That makes it a lot easier to read, you don't have to download and zip, you can just go to it straight from JIRA. Thanks!

          Show
          David Van Couvering added a comment - Hi, Sanket. Can you re-attach your documents as individual non-compressed files? That makes it a lot easier to read, you don't have to download and zip, you can just go to it straight from JIRA. Thanks!

            People

            • Assignee:
              John H. Embretsen
              Reporter:
              Sanket Sharma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development