Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1659

Add JMX support for dynamic reconfiguration

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.5.0
    • Fix Version/s: 3.5.0
    • Component/s: server
    • Labels:
      None

      Description

      We need to update JMX during reconfigurations. Currently, reconfiguration changes are not reflected in JConsole.

      1. ZK-1659-jmx console-after-remove-qp-reconfig.png
        30 kB
        Rakesh R
      2. ZK-1659-jmx console-before-reconfig.png
        32 kB
        Rakesh R
      3. ZOOKEEPER-1659.patch
        22 kB
        Rakesh R
      4. ZOOKEEPER-1659.patch
        21 kB
        Rakesh R
      5. ZOOKEEPER-1659.patch
        21 kB
        Rakesh R
      6. ZOOKEEPER-1659.patch
        12 kB
        Rakesh R
      7. ZOOKEEPER-1659.patch
        8 kB
        Rakesh R
      8. ZOOKEEPER-1659.patch
        7 kB
        Rakesh R

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in ZooKeeper-trunk #2325 (See https://builds.apache.org/job/ZooKeeper-trunk/2325/)
          ZOOKEEPER-1659. Add JMX support for dynamic reconfiguration (Rakesh R via michim) (michim: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1600481)

          • /zookeeper/trunk/CHANGES.txt
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/LocalPeerBean.java
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/LocalPeerMXBean.java
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/RemotePeerBean.java
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/RemotePeerMXBean.java
          • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java
          • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/JMXEnv.java
          • /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/ReconfigTest.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in ZooKeeper-trunk #2325 (See https://builds.apache.org/job/ZooKeeper-trunk/2325/ ) ZOOKEEPER-1659 . Add JMX support for dynamic reconfiguration (Rakesh R via michim) (michim: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1600481 ) /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/LocalPeerBean.java /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/LocalPeerMXBean.java /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/RemotePeerBean.java /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/quorum/RemotePeerMXBean.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/JMXEnv.java /zookeeper/trunk/src/java/test/org/apache/zookeeper/test/ReconfigTest.java
          Show
          michim Michi Mutsuzaki added a comment - http://svn.apache.org/viewvc?view=revision&revision=1600481
          Hide
          michim Michi Mutsuzaki added a comment -

          +1 I'm checking this in. Rakesh, thank you for taking care of this!

          Show
          michim Michi Mutsuzaki added a comment - +1 I'm checking this in. Rakesh, thank you for taking care of this!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12647957/ZOOKEEPER-1659.patch
          against trunk revision 1598087.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2121//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2121//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2121//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12647957/ZOOKEEPER-1659.patch against trunk revision 1598087. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2121//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2121//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2121//console This message is automatically generated.
          Hide
          shralex Alexander Shraer added a comment -

          +1 looks great!

          Show
          shralex Alexander Shraer added a comment - +1 looks great!
          Hide
          rakeshr Rakesh R added a comment -

          Attached new patch addressing the comments, please have a look at it. Thanks!

          Show
          rakeshr Rakesh R added a comment - Attached new patch addressing the comments, please have a look at it. Thanks!
          Hide
          shralex Alexander Shraer added a comment -

          Got it. thanks for the explanation and the screenshots! I think its very helpful.

          > int leavingIndex = 1;
          > int nextIndex = 2;

          I have one last comment - when removing/changing server 1 (for example), instead of checking that the remote bean for replica.1 was changed/removed only for ReplicatedServer_id2 (next on the list), could you please check that that the remote bean for replica.1 was changed/removed for both other servers ?

          Show
          shralex Alexander Shraer added a comment - Got it. thanks for the explanation and the screenshots! I think its very helpful. > int leavingIndex = 1; > int nextIndex = 2; I have one last comment - when removing/changing server 1 (for example), instead of checking that the remote bean for replica.1 was changed/removed only for ReplicatedServer_id2 (next on the list), could you please check that that the remote bean for replica.1 was changed/removed for both other servers ?
          Hide
          rakeshr Rakesh R added a comment -

          ffor remote bean:
          CommonNames.DOMAIN + ":name0=ReplicatedServer_id" + nextIndex + ",name1=replica." + changingIndex;
          for local bean: CommonNames.DOMAIN + ":name0=ReplicatedServer_id" + leavingIndex + ",name1=replica." + leavingIndex;

          Assume am having servers zk-1, zk-2, zk-3.

          For example, am checking the beans of zk-2 it has the following beans:
          1) localbean: name0=ReplicatedServer_id2,name1=replica.2
          2) remote bean of zk-1: name0=ReplicatedServer_id2,name1=replica.1
          3) remote bean of zk-3: name0=ReplicatedServer_id2,name1=replica.3

          the text here: https://zookeeper.apache.org/doc/r3.3.2/zookeeperJMX.html
          indicates that the indices for replica.x and ReplicatedServer_id.x are the same (myid of the server). Can you explain why in the case of remote bean the indices are different ? should the documentation change ?

          I think no issues in docs.

          also, did you have a chance to try it with a jmx console ?

          Attached snap of the jmx console. Also I tried adding few hints, please forgive if its confusing.

          Show
          rakeshr Rakesh R added a comment - ffor remote bean: CommonNames.DOMAIN + ":name0=ReplicatedServer_id" + nextIndex + ",name1=replica." + changingIndex; for local bean: CommonNames.DOMAIN + ":name0=ReplicatedServer_id" + leavingIndex + ",name1=replica." + leavingIndex; Assume am having servers zk-1, zk-2, zk-3. For example, am checking the beans of zk-2 it has the following beans: 1) localbean: name0=ReplicatedServer_id2,name1=replica.2 2) remote bean of zk-1: name0=ReplicatedServer_id2,name1=replica.1 3) remote bean of zk-3: name0=ReplicatedServer_id2,name1=replica.3 the text here: https://zookeeper.apache.org/doc/r3.3.2/zookeeperJMX.html indicates that the indices for replica.x and ReplicatedServer_id.x are the same (myid of the server). Can you explain why in the case of remote bean the indices are different ? should the documentation change ? I think no issues in docs. also, did you have a chance to try it with a jmx console ? Attached snap of the jmx console. Also I tried adding few hints, please forgive if its confusing.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12647862/ZK-1659-jmx%20console-after-remove-qp-reconfig.png
          against trunk revision 1598087.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2120//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12647862/ZK-1659-jmx%20console-after-remove-qp-reconfig.png against trunk revision 1598087. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2120//console This message is automatically generated.
          Hide
          shralex Alexander Shraer added a comment -

          Thanks Rakesh! For the second question I meant this:

          for remote bean:
          CommonNames.DOMAIN + ":name0=ReplicatedServer_id" + nextIndex + ",name1=replica." + changingIndex;

          for local bean: CommonNames.DOMAIN + ":name0=ReplicatedServer_id" + leavingIndex + ",name1=replica." + leavingIndex;

          the text here: https://zookeeper.apache.org/doc/r3.3.2/zookeeperJMX.html
          indicates that the indices for replica.x and ReplicatedServer_id.x are the same (myid of the server). Can you explain why in the case of remote bean the indices are different ? should the documentation change ?

          also, did you have a chance to try it with a jmx console ?

          Show
          shralex Alexander Shraer added a comment - Thanks Rakesh! For the second question I meant this: for remote bean: CommonNames.DOMAIN + ":name0=ReplicatedServer_id" + nextIndex + ",name1=replica." + changingIndex; for local bean: CommonNames.DOMAIN + ":name0=ReplicatedServer_id" + leavingIndex + ",name1=replica." + leavingIndex; the text here: https://zookeeper.apache.org/doc/r3.3.2/zookeeperJMX.html indicates that the indices for replica.x and ReplicatedServer_id.x are the same (myid of the server). Can you explain why in the case of remote bean the indices are different ? should the documentation change ? also, did you have a chance to try it with a jmx console ?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12647814/ZOOKEEPER-1659.patch
          against trunk revision 1598087.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2119//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2119//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2119//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12647814/ZOOKEEPER-1659.patch against trunk revision 1598087. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2119//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2119//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2119//console This message is automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          new tests verifyRemoteBean(String remoteBean) seems to be verifying that the bean is not registered. Please change the function name to reflect that.

          I've replaced using JMX.ensureNone(rBeanName). This is already available and serves the purpose.

          I don't fully understand the remote and local bean naming convention. Can you please explain ?

          It is actually to represents the object name pattern identifying the MBean to be retrieved, in our case it is RemotePeerMXBean and LocalPeerMXBean beans. Test case first tries to get the given MBean, then gets the value of a specific attribute of that MBean and does the assertion. I've modified the variable names 'remoteBean and localBean' to 'remotePeerBean and localPeerBean'. Also I've tried modifying the method names to get more clarity, is that fine ?

          Show
          rakeshr Rakesh R added a comment - new tests verifyRemoteBean(String remoteBean) seems to be verifying that the bean is not registered. Please change the function name to reflect that. I've replaced using JMX.ensureNone(rBeanName). This is already available and serves the purpose. I don't fully understand the remote and local bean naming convention. Can you please explain ? It is actually to represents the object name pattern identifying the MBean to be retrieved, in our case it is RemotePeerMXBean and LocalPeerMXBean beans. Test case first tries to get the given MBean, then gets the value of a specific attribute of that MBean and does the assertion. I've modified the variable names 'remoteBean and localBean' to 'remotePeerBean and localPeerBean'. Also I've tried modifying the method names to get more clarity, is that fine ?
          Hide
          shralex Alexander Shraer added a comment -

          Thanks for adding more tests! Looking at the new tests verifyRemoteBean(String remoteBean) seems to be verifying that the bean is not registered. Please change the function name to reflect that.

          I don't fully understand the remote and local bean naming convention. Can you please explain ?

          Show
          shralex Alexander Shraer added a comment - Thanks for adding more tests! Looking at the new tests verifyRemoteBean(String remoteBean) seems to be verifying that the bean is not registered. Please change the function name to reflect that. I don't fully understand the remote and local bean naming convention. Can you please explain ?
          Hide
          michim Michi Mutsuzaki added a comment -

          +1. Alex, could you also take a look at the latest patch?

          Show
          michim Michi Mutsuzaki added a comment - +1. Alex, could you also take a look at the latest patch?
          Hide
          rakeshr Rakesh R added a comment -

          Michi Mutsuzaki As per discussion with Alexander Shraer its decided to keep the localbean. I've attached new patch, could you please review it when you get sometime. Thanks!

          Show
          rakeshr Rakesh R added a comment - Michi Mutsuzaki As per discussion with Alexander Shraer its decided to keep the localbean. I've attached new patch, could you please review it when you get sometime. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12647430/ZOOKEEPER-1659.patch
          against trunk revision 1598087.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2117//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2117//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2117//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12647430/ZOOKEEPER-1659.patch against trunk revision 1598087. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2117//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2117//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2117//console This message is automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          Sometime back myself and Alex had a private discussion to understand more on the reconfig behaviours. Thanks Alexander Shraer. Here one thing come across is, even after removal of quorum peer from the configuration, the server will be running fine and old clients as well as new clients will work happily. Also, the QuorumPeer parameters are likely not affected either. Considering these cases, IMHO LocalPeerBean(self) can be accessible to the external users as long as the server is alive, what others opinion?.

          Following are the attributes of LocalPeerBean:

          • State
          • QuorumAddress
          • ElectionAddress
          • LearnerType
          • ClientAddress
          • ElectionType
          • TickTime
          • MaxClientCnxnsPerHost
          • MinSessionTimeout
          • MaxSessionTimeout
          • InitLimit
          • Tick
          • SyncLimit
          • Name
          • StartTime

          Now as part of this patch proposal we will be adding the following attributes too.

          • QuorumSystemInfo
          • ConfigVersion
          Show
          rakeshr Rakesh R added a comment - Sometime back myself and Alex had a private discussion to understand more on the reconfig behaviours. Thanks Alexander Shraer . Here one thing come across is, even after removal of quorum peer from the configuration, the server will be running fine and old clients as well as new clients will work happily. Also, the QuorumPeer parameters are likely not affected either. Considering these cases, IMHO LocalPeerBean(self) can be accessible to the external users as long as the server is alive, what others opinion?. Following are the attributes of LocalPeerBean: State QuorumAddress ElectionAddress LearnerType ClientAddress ElectionType TickTime MaxClientCnxnsPerHost MinSessionTimeout MaxSessionTimeout InitLimit Tick SyncLimit Name StartTime Now as part of this patch proposal we will be adding the following attributes too. QuorumSystemInfo ConfigVersion
          Hide
          michim Michi Mutsuzaki added a comment -

          ping!

          Show
          michim Michi Mutsuzaki added a comment - ping!
          Hide
          shralex Alexander Shraer added a comment -

          Thanks, Rakesh! The code for adding and removing remote peers is much clearer now too.
          But what about the local one ? did you decide to leave it always registered even if its removed from the ensemble ?

          Show
          shralex Alexander Shraer added a comment - Thanks, Rakesh! The code for adding and removing remote peers is much clearer now too. But what about the local one ? did you decide to leave it always registered even if its removed from the ensemble ?
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12646463/ZOOKEEPER-1659.patch
          against trunk revision 1596684.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2112//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2112//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2112//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12646463/ZOOKEEPER-1659.patch against trunk revision 1596684. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2112//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2112//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2112//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12646463/ZOOKEEPER-1659.patch
          against trunk revision 1596684.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2111//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2111//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2111//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12646463/ZOOKEEPER-1659.patch against trunk revision 1596684. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2111//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2111//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2111//console This message is automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          Thanks Flavio Junqueira, Alexander Shraer for the comments. Exposes new attribute LocalPeerMXBean#getQuorumSystemInfo(), which inturn make use of QuorumVerifier#toString() implementation. Kindly review the latest patch, Thanks!

          Following is the sample value of QuorumHierarchical's QuorumSystemInfo attribute value:

          server.1=127.0.0.1:12223:12228:participant;0.0.0.0:11233
          server.2=127.0.0.1:12224:12229:participant;0.0.0.0:11234
          server.3=127.0.0.1:12225:12230:participant;0.0.0.0:11235
          server.4=127.0.0.1:12226:12231:participant;0.0.0.0:11236
          server.5=127.0.0.1:12227:12232:participant;0.0.0.0:11237
          group.1=1:2:3
          group.2=4:5
          weight.1=1
          weight.2=1
          weight.3=1
          weight.4=0
          weight.5=0
          version=100000000
          
          Show
          rakeshr Rakesh R added a comment - Thanks Flavio Junqueira , Alexander Shraer for the comments. Exposes new attribute LocalPeerMXBean#getQuorumSystemInfo(), which inturn make use of QuorumVerifier#toString() implementation. Kindly review the latest patch, Thanks! Following is the sample value of QuorumHierarchical's QuorumSystemInfo attribute value: server.1=127.0.0.1:12223:12228:participant;0.0.0.0:11233 server.2=127.0.0.1:12224:12229:participant;0.0.0.0:11234 server.3=127.0.0.1:12225:12230:participant;0.0.0.0:11235 server.4=127.0.0.1:12226:12231:participant;0.0.0.0:11236 server.5=127.0.0.1:12227:12232:participant;0.0.0.0:11237 group.1=1:2:3 group.2=4:5 weight.1=1 weight.2=1 weight.3=1 weight.4=0 weight.5=0 version=100000000
          Hide
          shralex Alexander Shraer added a comment -

          If I understand correctly currently the QuorumServers are exported not QuorumVerifier.

          Show
          shralex Alexander Shraer added a comment - If I understand correctly currently the QuorumServers are exported not QuorumVerifier.
          Hide
          fpj Flavio Junqueira added a comment -

          We could easily add a method to the QuorumVerifier interface that returns a string with the quorum system information (e.g., getQuorumSystemInfo()), and use it with JMX, no?

          Show
          fpj Flavio Junqueira added a comment - We could easily add a method to the QuorumVerifier interface that returns a string with the quorum system information (e.g., getQuorumSystemInfo()), and use it with JMX, no?
          Hide
          rakeshr Rakesh R added a comment -

          Attaching new patch where it will identify the diff as follows. With this it won't do unnecessary un-registration of the existing beans and avoided the possibility of above mentioned exceptions.

          1. new members will register their beans
          2. existing members will update their QS instance
          3. leaving members will be unregistered
          Show
          rakeshr Rakesh R added a comment - Attaching new patch where it will identify the diff as follows. With this it won't do unnecessary un-registration of the existing beans and avoided the possibility of above mentioned exceptions. new members will register their beans existing members will update their QS instance leaving members will be unregistered
          Hide
          otis Otis Gospodnetic added a comment -

          +1 for making sure changes are backwards-compatible. We monitor Zookeeper with SPM and would love to be able to use the same agent for multiple/all ZK versions instead of having ZK version-specific agents.

          Show
          otis Otis Gospodnetic added a comment - +1 for making sure changes are backwards-compatible. We monitor Zookeeper with SPM and would love to be able to use the same agent for multiple/all ZK versions instead of having ZK version-specific agents.
          Hide
          rakeshr Rakesh R added a comment -

          After unregister if anyone(for ex: monitoring tool) queried to get the attribute value, it would occur the following exception.

          javax.management.InstanceNotFoundException: org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Leader,name3=InMemoryDataTree
          	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
          
          Show
          rakeshr Rakesh R added a comment - After unregister if anyone(for ex: monitoring tool) queried to get the attribute value, it would occur the following exception. javax.management.InstanceNotFoundException: org.apache.ZooKeeperService:name0=ReplicatedServer_id3,name1=replica.3,name2=Leader,name3=InMemoryDataTree at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
          Hide
          shralex Alexander Shraer added a comment -

          Doing the diff (but simplifying the diff code a bit, perhaps as I suggested above) or unregistering/re-registering everything both sound fine to me, but I've never used JMX so not sure what's the implications of unregistering/re-registering something. So either way you choose I'd suggest to run the JMX console and see how it works during reconfig.

          Regarding exposing the Quorum System in JMX, I'm ok with postponing this until after 3.5.0, Flavio Junqueira are you fine with that ?

          Show
          shralex Alexander Shraer added a comment - Doing the diff (but simplifying the diff code a bit, perhaps as I suggested above) or unregistering/re-registering everything both sound fine to me, but I've never used JMX so not sure what's the implications of unregistering/re-registering something. So either way you choose I'd suggest to run the JMX console and see how it works during reconfig. Regarding exposing the Quorum System in JMX, I'm ok with postponing this until after 3.5.0, Flavio Junqueira are you fine with that ?
          Hide
          rakeshr Rakesh R added a comment -

          Thanks Alexander Shraer and Flavio Junqueira

          removing a server may not lead to shutting it down - it may be an intermediate step for changing its role (there are cases where this is necessary when changing an observer to a follower - an observer can't vote so if its vote is needed to establish the new config we may need to first remove it so that its a non-voting follower).

          Yeah this is good point, I missed it. Initially I was thinking to identify only the changed QPs and tried to unregister/re-register it. Now after seeing this scenario I've diff idea in my mind. How about un-register all the remote beans and re-register back according to the new QuorumVerifier#getAllMembers() ?. Here one concern is, all the remote QPs will be unregistered for a moment until its re-registration, irrespective of QuorumServer has configuration changes or not. With this approach, code will be simple rather than having complex logic for diff and identifying only those changed QPs.

          ZK JMX hierarchy is:

          • jmxQuorumBean (himself registered as parent bean)
            • LocalPeerBean (self quorumpeer)
            • RemotePeerBean (for each remote qp in the config)

          One thing this patch is not handling is quorum system changes. if I understand correctly if we switch from majority to hierarchical or the other way around, or change parameters of a hierarchical quorum system, this won't be reflected in JMX that only has the QuorumServers

          Presently Hierarchical system info is not exposed and not available in any of the existing jmx beans. I also feel, this can be exposed after 3.5.0.

          Show
          rakeshr Rakesh R added a comment - Thanks Alexander Shraer and Flavio Junqueira removing a server may not lead to shutting it down - it may be an intermediate step for changing its role (there are cases where this is necessary when changing an observer to a follower - an observer can't vote so if its vote is needed to establish the new config we may need to first remove it so that its a non-voting follower). Yeah this is good point, I missed it. Initially I was thinking to identify only the changed QPs and tried to unregister/re-register it. Now after seeing this scenario I've diff idea in my mind. How about un-register all the remote beans and re-register back according to the new QuorumVerifier#getAllMembers() ?. Here one concern is, all the remote QPs will be unregistered for a moment until its re-registration, irrespective of QuorumServer has configuration changes or not. With this approach, code will be simple rather than having complex logic for diff and identifying only those changed QPs. ZK JMX hierarchy is: jmxQuorumBean (himself registered as parent bean) LocalPeerBean (self quorumpeer) RemotePeerBean (for each remote qp in the config) One thing this patch is not handling is quorum system changes. if I understand correctly if we switch from majority to hierarchical or the other way around, or change parameters of a hierarchical quorum system, this won't be reflected in JMX that only has the QuorumServers Presently Hierarchical system info is not exposed and not available in any of the existing jmx beans. I also feel, this can be exposed after 3.5.0.
          Hide
          fpj Flavio Junqueira added a comment -

          I'm aware of deployments that use HQS. However, the main general concern I have is that the other day I tried to implement a new quorum verifier and a lot of tests failed. It seems that there are parts of the code base, especially tests, that are hardcoding majorities, which I find a bit annoying. It would be great if we could keep the use of quorums independent of the actual implementation, and in this particular case rely on the QuorumVerifier interface.

          Show
          fpj Flavio Junqueira added a comment - I'm aware of deployments that use HQS. However, the main general concern I have is that the other day I tried to implement a new quorum verifier and a lot of tests failed. It seems that there are parts of the code base, especially tests, that are hardcoding majorities, which I find a bit annoying. It would be great if we could keep the use of quorums independent of the actual implementation, and in this particular case rely on the QuorumVerifier interface.
          Hide
          shralex Alexander Shraer added a comment -

          One thing this patch is not handling is quorum system changes. if I understand correctly if we switch from majority to hierarchical or the other way around, or change parameters of a hierarchical quorum system, this won't be reflected in JMX that only has the QuorumServers. I'm not sure this is such a major concern since I don't even know if anyone uses the hierarchical quorum system. Flavio Junqueira what do you think ?

          If Quorum System info is not added to JMX in this patch, we should open a jira to do this in the future, after 3.5.0.

          Show
          shralex Alexander Shraer added a comment - One thing this patch is not handling is quorum system changes. if I understand correctly if we switch from majority to hierarchical or the other way around, or change parameters of a hierarchical quorum system, this won't be reflected in JMX that only has the QuorumServers. I'm not sure this is such a major concern since I don't even know if anyone uses the hierarchical quorum system. Flavio Junqueira what do you think ? If Quorum System info is not added to JMX in this patch, we should open a jira to do this in the future, after 3.5.0.
          Hide
          shralex Alexander Shraer added a comment -

          One more small comment - in the beginning of updateMXBeans, I suggest to simplify by having just 2 sets - joiningMembers and leavingMembers.

          Set<Long> joiningMembers = new HashSet<Long>(allMembers.keySet()).removeAll(remotePeerBeans.keySet());
          Set<Long> leavingMembers = new HashSet<Long>(remotePeerBeans.keySet()).removeAll(allMembers.keySet());

          note sure about removing getId() from these - see my previous comment.

          maybe it'll be also good to rename the parameter allMembers to newMembers.

          Show
          shralex Alexander Shraer added a comment - One more small comment - in the beginning of updateMXBeans, I suggest to simplify by having just 2 sets - joiningMembers and leavingMembers. Set<Long> joiningMembers = new HashSet<Long>(allMembers.keySet()).removeAll(remotePeerBeans.keySet()); Set<Long> leavingMembers = new HashSet<Long>(remotePeerBeans.keySet()).removeAll(allMembers.keySet()); note sure about removing getId() from these - see my previous comment. maybe it'll be also good to rename the parameter allMembers to newMembers.
          Hide
          shralex Alexander Shraer added a comment -

          I'd like to go over this too - probably over the weekend. Sorry for the delay, Rakesh!

          > "I've one question, what will happen to the qp server which is leaving the quorum ?
          > In the patch it will not remove the self mxbean, considering that admin will make it down and during that time the jmx
          > beans will get automatically unregistered. Is that fine ?"

          Intuitively it'd be good if all the places listing the configuration are kept consistent. So I'd say that the server should remove itself from this list so that the list in the config znode and in the JMX are consistent. Also - removing a server may not lead to shutting it down - it may be an intermediate step for changing its role (there are cases where this is necessary when changing an observer to a follower - an observer can't vote so if its vote is needed to establish the new config we may need to first remove it so that its a non-voting follower).

          Show
          shralex Alexander Shraer added a comment - I'd like to go over this too - probably over the weekend. Sorry for the delay, Rakesh! > "I've one question, what will happen to the qp server which is leaving the quorum ? > In the patch it will not remove the self mxbean, considering that admin will make it down and during that time the jmx > beans will get automatically unregistered. Is that fine ?" Intuitively it'd be good if all the places listing the configuration are kept consistent. So I'd say that the server should remove itself from this list so that the list in the config znode and in the JMX are consistent. Also - removing a server may not lead to shutting it down - it may be an intermediate step for changing its role (there are cases where this is necessary when changing an observer to a follower - an observer can't vote so if its vote is needed to establish the new config we may need to first remove it so that its a non-voting follower).
          Hide
          michim Michi Mutsuzaki added a comment -

          +1 The patch looks good to me. It would be great if somebody more familiar with JMX code can review this patch.

          Show
          michim Michi Mutsuzaki added a comment - +1 The patch looks good to me. It would be great if somebody more familiar with JMX code can review this patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12644408/ZOOKEEPER-1659.patch
          against trunk revision 1593682.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2088//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2088//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2088//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644408/ZOOKEEPER-1659.patch against trunk revision 1593682. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2088//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2088//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2088//console This message is automatically generated.
          Hide
          rakeshr Rakesh R added a comment -

          Thank you Alexander Shraer for the inputs. I've tried an initial draft patch where it has the following changes:

          1. Remove the remote.replica.bean from the ZKService bean for the leaving servers
          2. Add the new remote.replica.bean to the ZKService bean for the joining servers
          3. New Attributes for each remote.replica.bean
            -> ElectionAddress
            -> ClientAddress
            -> LearnerType
          4. New Attributes for each local.replica.bean added few more attributes
            -> ElectionAddress
            -> ClientAddress
            -> LearnerType
            -> ConfigVersion

          I've one question, what will happen to the qp server which is leaving the quorum ?
          In the patch it will not remove the self mxbean, considering that admin will make it down and during that time the jmx beans will get automatically unregistered. Is that fine ?

          Thanks,
          Rakesh

          Show
          rakeshr Rakesh R added a comment - Thank you Alexander Shraer for the inputs. I've tried an initial draft patch where it has the following changes: Remove the remote.replica.bean from the ZKService bean for the leaving servers Add the new remote.replica.bean to the ZKService bean for the joining servers New Attributes for each remote.replica.bean -> ElectionAddress -> ClientAddress -> LearnerType New Attributes for each local.replica.bean added few more attributes -> ElectionAddress -> ClientAddress -> LearnerType -> ConfigVersion I've one question, what will happen to the qp server which is leaving the quorum ? In the patch it will not remove the self mxbean, considering that admin will make it down and during that time the jmx beans will get automatically unregistered. Is that fine ? Thanks, Rakesh
          Hide
          michim Michi Mutsuzaki added a comment -

          This is marked as a blocker, but there is nobody assigned to it. Any volunteers?

          Show
          michim Michi Mutsuzaki added a comment - This is marked as a blocker, but there is nobody assigned to it. Any volunteers?
          Hide
          shralex Alexander Shraer added a comment -

          Rakesh, the committed config is stored in QuorumPeer.quorumVerifier, and set in QuorumPeer.setQuorumVerifier(), so that may be a good place to start. A QuorumVerifier can currently be either a QurumMaj or QuorumHierarchical, see org.apache.zookeeper.server.quorum.flexible.

          If you have any questions about the reconfig implementation, feel free to ping me shralex@gmail.com

          Show
          shralex Alexander Shraer added a comment - Rakesh, the committed config is stored in QuorumPeer.quorumVerifier, and set in QuorumPeer.setQuorumVerifier(), so that may be a good place to start. A QuorumVerifier can currently be either a QurumMaj or QuorumHierarchical, see org.apache.zookeeper.server.quorum.flexible. If you have any questions about the reconfig implementation, feel free to ping me shralex@gmail.com
          Hide
          rakeshr Rakesh R added a comment -

          Alexander Shraer Thanks for the reply, basically first it need to identify the necessary attributes which can be exposed in MBeans. I'll go through the reconfig implementation this weekend and hope it would give some more idea.

          Patrick Hunt It would be great if you can add your thoughts when you get sometime, it would help us to set the direction. Thanks.

          Hi Edward Ribeiro,

          See https://issues.apache.org/jira/browse/ZOOKEEPER-1423 for further details. This patch probably doesn't apply cleanly anymore, but at least you may see some starting points. PS: a knowledgeable JMX opinion about my patch would be highly appreciated

          I've seen your patch and added my comments on it. Please see it.

          Show
          rakeshr Rakesh R added a comment - Alexander Shraer Thanks for the reply, basically first it need to identify the necessary attributes which can be exposed in MBeans. I'll go through the reconfig implementation this weekend and hope it would give some more idea. Patrick Hunt It would be great if you can add your thoughts when you get sometime, it would help us to set the direction. Thanks. Hi Edward Ribeiro , See https://issues.apache.org/jira/browse/ZOOKEEPER-1423 for further details. This patch probably doesn't apply cleanly anymore, but at least you may see some starting points. PS: a knowledgeable JMX opinion about my patch would be highly appreciated I've seen your patch and added my comments on it. Please see it.
          Hide
          eribeiro Edward Ribeiro added a comment -

          Hi guys,

          Although I am far from being a JMX expert, I have some interest on this issue, and had submitted a patch to add a 'dir' command available from both 4lw and JMX. See https://issues.apache.org/jira/browse/ZOOKEEPER-1423 for further details. This patch probably doesn't apply cleanly anymore, but at least you may see some starting points. PS: a knowledgeable JMX opinion about my patch would be highly appreciated.

          Show
          eribeiro Edward Ribeiro added a comment - Hi guys, Although I am far from being a JMX expert, I have some interest on this issue, and had submitted a patch to add a 'dir' command available from both 4lw and JMX. See https://issues.apache.org/jira/browse/ZOOKEEPER-1423 for further details. This patch probably doesn't apply cleanly anymore, but at least you may see some starting points. PS: a knowledgeable JMX opinion about my patch would be highly appreciated.
          Hide
          shralex Alexander Shraer added a comment -

          Hi Rakesh,

          To tell you the truth I've never used JMX, and not familiar with ZooKeeper support for this, so Patrick Hunt may have a better idea of what we need here. My guess is that we'd want to at least see the latest committed dynamic config, even better would be to know what it is at different servers so that we know if some server has a stale config. With regard to in-progress reconfigs, I'm not sure how this would work since they only take a few ms at most (as any other op). State transfer happens before the reconfig is invoked. Perhaps reconfig proposal and commit times can be recorded, but this seems lower priority.

          It would be good to monitor the set of servers connected/synced to the leader and their roles (some of them will be paticipants/observers in the current config and others may be non-voting followers, that don't belong to the current config). This way an operator can know that server A is already in-sync with the leader and he can invoke reconfig to add A.

          In general we probably want to add some minimal support first so that its no longer a blocker for 3.5.0, and then potentially add more info if needed.

          Alex

          Show
          shralex Alexander Shraer added a comment - Hi Rakesh, To tell you the truth I've never used JMX, and not familiar with ZooKeeper support for this, so Patrick Hunt may have a better idea of what we need here. My guess is that we'd want to at least see the latest committed dynamic config, even better would be to know what it is at different servers so that we know if some server has a stale config. With regard to in-progress reconfigs, I'm not sure how this would work since they only take a few ms at most (as any other op). State transfer happens before the reconfig is invoked. Perhaps reconfig proposal and commit times can be recorded, but this seems lower priority. It would be good to monitor the set of servers connected/synced to the leader and their roles (some of them will be paticipants/observers in the current config and others may be non-voting followers, that don't belong to the current config). This way an operator can know that server A is already in-sync with the leader and he can invoke reconfig to add A. In general we probably want to add some minimal support first so that its no longer a blocker for 3.5.0, and then potentially add more info if needed. Alex
          Hide
          rakeshr Rakesh R added a comment -

          Hi Alexander Shraer, Could you please give some more insight into the description, let me try to understand and would like to take this ahead.
          I've few queries:
          1) Intention is to introduce new reconfig attributes like:

          • config version number
          • new members added at previous reconfig call
          • old members removed at previous reconfig call
          • shows if any reconfig is in progress et.

          2) Impacted any of the existing JMX attributes due to the reconfigurations ?

          Thanks,
          Rakesh

          Show
          rakeshr Rakesh R added a comment - Hi Alexander Shraer , Could you please give some more insight into the description, let me try to understand and would like to take this ahead. I've few queries: 1) Intention is to introduce new reconfig attributes like: config version number new members added at previous reconfig call old members removed at previous reconfig call shows if any reconfig is in progress et. 2) Impacted any of the existing JMX attributes due to the reconfigurations ? Thanks, Rakesh
          Hide
          phunt Patrick Hunt added a comment -

          We need to triage this at a minimum. Also effects on 4lw?

          Show
          phunt Patrick Hunt added a comment - We need to triage this at a minimum. Also effects on 4lw?
          Hide
          shralex Alexander Shraer added a comment -

          Would be good if someone experienced with JMX can take this.

          Show
          shralex Alexander Shraer added a comment - Would be good if someone experienced with JMX can take this.

            People

            • Assignee:
              rakeshr Rakesh R
              Reporter:
              shralex Alexander Shraer
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development