Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13782

Make MutableRates metrics thread-local write, aggregate-on-read

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
    • Component/s: metrics
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently the MutableRates metrics class serializes all writes to metrics it contains because of its use of MetricsRegistry.add() (i.e., even two increments of unrelated metrics contained within the same MutableRates object will serialize w.r.t. each other). This class is used by RpcDetailedMetrics, which may have many hundreds of threads contending to modify these metrics. Instead we should allow updates to unrelated metrics objects to happen concurrently. To do so we can let each thread locally collect metrics, and on a snapshot, aggregate the metrics from all of the threads.

      I have collected some benchmark performance numbers in HADOOP-13747 (https://issues.apache.org/jira/secure/attachment/12835043/benchmark_results) which indicate that this can bring significantly higher performance in high contention situations.

      1. HADOOP-13782.000.patch
        17 kB
        Erik Krogen
      2. HADOOP-13782.001.patch
        17 kB
        Erik Krogen
      3. HADOOP-13782.002.patch
        15 kB
        Erik Krogen
      4. HADOOP-13782.003.patch
        16 kB
        Erik Krogen
      5. HADOOP-13782.004.patch
        15 kB
        Erik Krogen
      6. HADOOP-13782.005.patch
        15 kB
        Erik Krogen
      7. HADOOP-13782.006.patch
        16 kB
        Erik Krogen

        Issue Links

          Activity

          Hide
          xkrogen Erik Krogen added a comment -

          Attaching v000 patch.

          We have decided to create a new class, MutableRatesWithAggregation, rather than modify the existing MutableRates metrics class since it is an Evolving interface which can only be changed at a minor version bump, to be able to backport to older versions. I have aded a comment on MutableRates noting that its performance in high contention situations will be lacking in comparison.

          I have updated RpcDetailedMetrics, the only consumer of MutableRates, to use MutableRatesWithAggregation instead since it is a point of potentially very high contention.

          Show
          xkrogen Erik Krogen added a comment - Attaching v000 patch. We have decided to create a new class, MutableRatesWithAggregation , rather than modify the existing MutableRates metrics class since it is an Evolving interface which can only be changed at a minor version bump, to be able to backport to older versions. I have aded a comment on MutableRates noting that its performance in high contention situations will be lacking in comparison. I have updated RpcDetailedMetrics , the only consumer of MutableRates , to use MutableRatesWithAggregation instead since it is a point of potentially very high contention.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 13s trunk passed
          +1 compile 7m 4s trunk passed
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 23s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 39s the patch passed
          +1 compile 8m 13s the patch passed
          +1 javac 8m 13s the patch passed
          -0 checkstyle 0m 23s hadoop-common-project/hadoop-common: The patch generated 16 new + 21 unchanged - 2 fixed = 37 total (was 23)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 41s the patch passed
          +1 javadoc 0m 47s the patch passed
          -1 unit 8m 1s hadoop-common in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          40m 53s



          Reason Tests
          Failed junit tests hadoop.metrics2.lib.TestMutableMetrics
            hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13782
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836971/HADOOP-13782.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 78b833a17407 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7534aee
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10982/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10982/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10982/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10982/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 13s trunk passed +1 compile 7m 4s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 23s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 8m 13s the patch passed +1 javac 8m 13s the patch passed -0 checkstyle 0m 23s hadoop-common-project/hadoop-common: The patch generated 16 new + 21 unchanged - 2 fixed = 37 total (was 23) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 41s the patch passed +1 javadoc 0m 47s the patch passed -1 unit 8m 1s hadoop-common in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 40m 53s Reason Tests Failed junit tests hadoop.metrics2.lib.TestMutableMetrics   hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13782 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836971/HADOOP-13782.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 78b833a17407 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7534aee Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10982/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10982/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10982/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10982/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Seems checkstyle does not like lambda syntax... Attaching v001 patch with test and checkstyle fixes.

          Leaving two checkstyle points unaddressed:
          1. Complains about MutableRatesWithAggregation.rates not being private, but this is standard for @Metric metrics.
          2. Complains about MutableRatesWithAggregation.java:60 being long. I'm hesitant to break up a type statement; looks pretty ugly if I do.

          Show
          xkrogen Erik Krogen added a comment - Seems checkstyle does not like lambda syntax... Attaching v001 patch with test and checkstyle fixes. Leaving two checkstyle points unaddressed: 1. Complains about MutableRatesWithAggregation.rates not being private, but this is standard for @Metric metrics. 2. Complains about MutableRatesWithAggregation.java:60 being long. I'm hesitant to break up a type statement; looks pretty ugly if I do.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 55s trunk passed
          +1 compile 7m 42s trunk passed
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 35s trunk passed
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 7m 55s the patch passed
          +1 javac 7m 55s the patch passed
          -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 2 new + 21 unchanged - 2 fixed = 23 total (was 23)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 34s the patch passed
          +1 javadoc 0m 48s the patch passed
          +1 unit 8m 29s hadoop-common in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          42m 57s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13782
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836990/HADOOP-13782.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 86fe8d508b4e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5cad93d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10985/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10985/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10985/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 55s trunk passed +1 compile 7m 42s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 35s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 7m 55s the patch passed +1 javac 7m 55s the patch passed -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 2 new + 21 unchanged - 2 fixed = 23 total (was 23) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 8m 29s hadoop-common in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 42m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13782 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836990/HADOOP-13782.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 86fe8d508b4e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5cad93d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10985/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10985/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10985/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for the patch. LGTM overall. A few detailed comments:

          1. It'd be ideal if we can simplify the two internal classes LocalMutableRate and MutableRateInternal, and also better fit them with existing MutableStat or MutableRate classes. We discussed offline an issue in existing MutableStat batch add method around intervalStat. I think we should document the issue so other developers understand the motivation of creating a simpler rate class.
          2. The below synchronization behavior is different than MutableStat, where both snapshot and add methods are synchronized. Should we allow thread-local add while one thread is doing snapshot?
              @Override
              public void snapshot(MetricsRecordBuilder rb, boolean all) {
                synchronized (globalMetrics) {
            
          3. Maybe we should comment below that we will be doing aggregation (a main logic in this class)
            } else {
                      for (Map.Entry<String, LocalMutableRate> ent : map.entrySet()) {
            
          4. Cosmetic: since getLocalMetrics is short and is only used by add (which itself is short), can we merge the two methods?
          5. Cosmetic: as a follow-on we can consider consolidating the old MutableRates and the new MutableRatesWithAggregation to reduce duplication
          Show
          zhz Zhe Zhang added a comment - Thanks Erik for the patch. LGTM overall. A few detailed comments: It'd be ideal if we can simplify the two internal classes LocalMutableRate and MutableRateInternal , and also better fit them with existing MutableStat or MutableRate classes. We discussed offline an issue in existing MutableStat batch add method around intervalStat . I think we should document the issue so other developers understand the motivation of creating a simpler rate class. The below synchronization behavior is different than MutableStat , where both snapshot and add methods are synchronized . Should we allow thread-local add while one thread is doing snapshot ? @Override public void snapshot(MetricsRecordBuilder rb, boolean all) { synchronized (globalMetrics) { Maybe we should comment below that we will be doing aggregation (a main logic in this class) } else { for (Map.Entry< String , LocalMutableRate> ent : map.entrySet()) { Cosmetic: since getLocalMetrics is short and is only used by add (which itself is short), can we merge the two methods? Cosmetic: as a follow-on we can consider consolidating the old MutableRates and the new MutableRatesWithAggregation to reduce duplication
          Hide
          xkrogen Erik Krogen added a comment -

          Zhe Zhang, thanks for the review.

          1. Yes, it would be ideal. I've opened HADOOP-13804 to address the issue with MutableStat which was causing it to be unfit to use here. Fixing this issue will enable us to remove both inner classes.
          2. We want to ensure that add is a non-synchronized method to allow for concurrency (otherwise we would defeat the advantages of this class over MutableRates). A snapshot will not present a point-in-time consistent snapshot across all threads since that would require a higher degree of synchronization, but it does not seem that this should be necessary given that the values are just metrics and the only inexactness is exactly which side of a snapshot a metric's addition appears on (i.e. no values will be lost).
          3. Sure.
          4. getLocalMetrics was originally used in two places which is why I had it has a separate method. Good catch that this is no longer necessary.
          5. Agreed that this would be ideal.

          I'm attaching a new patch, v002, addressing these changes but unmarking patch available since these changes rely on the patch in HADOOP-13804.

          Show
          xkrogen Erik Krogen added a comment - Zhe Zhang , thanks for the review. 1. Yes, it would be ideal. I've opened HADOOP-13804 to address the issue with MutableStat which was causing it to be unfit to use here. Fixing this issue will enable us to remove both inner classes. 2. We want to ensure that add is a non-synchronized method to allow for concurrency (otherwise we would defeat the advantages of this class over MutableRates ). A snapshot will not present a point-in-time consistent snapshot across all threads since that would require a higher degree of synchronization, but it does not seem that this should be necessary given that the values are just metrics and the only inexactness is exactly which side of a snapshot a metric's addition appears on (i.e. no values will be lost). 3. Sure. 4. getLocalMetrics was originally used in two places which is why I had it has a separate method. Good catch that this is no longer necessary. 5. Agreed that this would be ideal. I'm attaching a new patch, v002, addressing these changes but unmarking patch available since these changes rely on the patch in HADOOP-13804 .
          Hide
          xkrogen Erik Krogen added a comment -

          Remarking as patch available since HADOOP-13804 has been committed.

          Show
          xkrogen Erik Krogen added a comment - Remarking as patch available since HADOOP-13804 has been committed.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 52s trunk passed
          +1 compile 11m 38s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 13s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 9m 48s the patch passed
          +1 javac 9m 48s the patch passed
          -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 4 new + 21 unchanged - 2 fixed = 25 total (was 23)
          +1 mvnsite 1m 15s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 50s the patch passed
          +1 javadoc 0m 53s the patch passed
          +1 unit 8m 1s hadoop-common in the patch passed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          50m 47s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13782
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837857/HADOOP-13782.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2d9fcc4173b9 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3dbad5d
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11017/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11017/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11017/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 52s trunk passed +1 compile 11m 38s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 13s trunk passed +1 mvneclipse 0m 23s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 9m 48s the patch passed +1 javac 9m 48s the patch passed -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 4 new + 21 unchanged - 2 fixed = 25 total (was 23) +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 8m 1s hadoop-common in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 50m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13782 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837857/HADOOP-13782.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2d9fcc4173b9 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3dbad5d Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11017/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11017/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11017/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Attaching v003 patch removing some unused imports.

          Show
          xkrogen Erik Krogen added a comment - Attaching v003 patch removing some unused imports.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 41s trunk passed
          -1 compile 6m 49s root in trunk failed.
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 48s trunk passed
          -1 mvninstall 0m 11s hadoop-common in the patch failed.
          -1 compile 0m 24s root in the patch failed.
          -1 javac 0m 24s root in the patch failed.
          -0 checkstyle 0m 21s hadoop-common-project/hadoop-common: The patch generated 5 new + 21 unchanged - 3 fixed = 26 total (was 24)
          -1 mvnsite 0m 11s hadoop-common in the patch failed.
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 11s hadoop-common in the patch failed.
          -1 javadoc 0m 10s hadoop-common in the patch failed.
          -1 unit 0m 11s hadoop-common in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          22m 18s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13782
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838003/HADOOP-13782.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6d99c978c70c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbb133c
          Default Java 1.8.0_111
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/branch-compile-root.txt
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 41s trunk passed -1 compile 6m 49s root in trunk failed. +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 48s trunk passed -1 mvninstall 0m 11s hadoop-common in the patch failed. -1 compile 0m 24s root in the patch failed. -1 javac 0m 24s root in the patch failed. -0 checkstyle 0m 21s hadoop-common-project/hadoop-common: The patch generated 5 new + 21 unchanged - 3 fixed = 26 total (was 24) -1 mvnsite 0m 11s hadoop-common in the patch failed. +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 11s hadoop-common in the patch failed. -1 javadoc 0m 10s hadoop-common in the patch failed. -1 unit 0m 11s hadoop-common in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 22m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13782 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838003/HADOOP-13782.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6d99c978c70c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbb133c Default Java 1.8.0_111 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11022/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Some extra characters snuck into MutableRate.java in the v003 patch... Attaching v004 patch to remedy.

          Show
          xkrogen Erik Krogen added a comment - Some extra characters snuck into MutableRate.java in the v003 patch... Attaching v004 patch to remedy.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 5s trunk passed
          +1 compile 12m 19s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 12s trunk passed
          +1 mvneclipse 0m 21s trunk passed
          +1 findbugs 1m 38s trunk passed
          +1 javadoc 0m 57s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 11m 9s the patch passed
          +1 javac 11m 9s the patch passed
          -0 checkstyle 0m 35s hadoop-common-project/hadoop-common: The patch generated 2 new + 21 unchanged - 2 fixed = 23 total (was 23)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 59s the patch passed
          +1 javadoc 0m 58s the patch passed
          +1 unit 10m 19s hadoop-common in the patch passed.
          +1 asflicense 0m 44s The patch does not generate ASF License warnings.
          55m 34s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13782
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838017/HADOOP-13782.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cff25740c5bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbb133c
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11025/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11025/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11025/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 5s trunk passed +1 compile 12m 19s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 21s trunk passed +1 findbugs 1m 38s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 11m 9s the patch passed +1 javac 11m 9s the patch passed -0 checkstyle 0m 35s hadoop-common-project/hadoop-common: The patch generated 2 new + 21 unchanged - 2 fixed = 23 total (was 23) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 58s the patch passed +1 unit 10m 19s hadoop-common in the patch passed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 55m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13782 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838017/HADOOP-13782.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cff25740c5bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbb133c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11025/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11025/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11025/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for the update! With HADOOP-13804 change the new class is much cleaner

          The current concurrency model is still a little complicated. snapshot has a nested synchronization on globalMetrics and stat, where stat is a local variable. Maybe we can simplify the concurrency model by:

          1. Make globalMetrics a ConcurrentMap
          2. Do we want to support multiple threads doing snapshot at the same time? If not, we should probably make it a synchronized method so it's easier to maintain and reason about
          3. Maybe creating a concurrent version of SampleStat, because that's the only object we want to protect from concurrent updating (local thread adding, and the snapshotting thread resetting).
              private class ConcurrentSampleStat extends SampleStat {
                @Override
                public synchronized void reset(){
                  super.reset();
                }
                @Override
                public synchronized SampleStat add(double x) {
                  return super.add(x);
                }
              }
            
          4. threadLocalMetricsMap can be a regular instead of concurrent map?

          Also, IIUC, snapshot is supposed to clear all metrics from the last window. In the v4 patch, if a certain type of metrics appeared in the last window but disappears in the current window (e.g. thread dies), the entry in globalMetrics is not cleared.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik for the update! With HADOOP-13804 change the new class is much cleaner The current concurrency model is still a little complicated. snapshot has a nested synchronization on globalMetrics and stat , where stat is a local variable. Maybe we can simplify the concurrency model by: Make globalMetrics a ConcurrentMap Do we want to support multiple threads doing snapshot at the same time? If not, we should probably make it a synchronized method so it's easier to maintain and reason about Maybe creating a concurrent version of SampleStat , because that's the only object we want to protect from concurrent updating (local thread adding, and the snapshotting thread resetting). private class ConcurrentSampleStat extends SampleStat { @Override public synchronized void reset(){ super .reset(); } @Override public synchronized SampleStat add( double x) { return super .add(x); } } threadLocalMetricsMap can be a regular instead of concurrent map? Also, IIUC, snapshot is supposed to clear all metrics from the last window. In the v4 patch, if a certain type of metrics appeared in the last window but disappears in the current window (e.g. thread dies), the entry in globalMetrics is not cleared.
          Hide
          xkrogen Erik Krogen added a comment -

          Thanks for the comments Zhe. I made snapshot/init synchronized methods; there wasn't really a reason they needed to synchronize on globalMetrics rather than just being synchronized methods. I also created ThreadSafeSampleStat; note that since the collection of the numSamples/total values and the reset should be done as one atomic step, I decided to create a wrapper class rather than a subclass. threadLocalMetricsMap needs to be a concurrent map since a snapshotting thread may be reading it while the local thread is doing a put operation.

          Good catch on the snapshot behavior not fully clearing; the correct behavior was lost between the v001 and v002 patches. I integrated the correct logic into ThreadSafeSampleStat.

          Attaching v005 patch.

          Show
          xkrogen Erik Krogen added a comment - Thanks for the comments Zhe. I made snapshot / init synchronized methods; there wasn't really a reason they needed to synchronize on globalMetrics rather than just being synchronized methods. I also created ThreadSafeSampleStat ; note that since the collection of the numSamples / total values and the reset should be done as one atomic step, I decided to create a wrapper class rather than a subclass. threadLocalMetricsMap needs to be a concurrent map since a snapshotting thread may be reading it while the local thread is doing a put operation. Good catch on the snapshot behavior not fully clearing; the correct behavior was lost between the v001 and v002 patches. I integrated the correct logic into ThreadSafeSampleStat . Attaching v005 patch.
          Hide
          xkrogen Erik Krogen added a comment -

          Attaching v006 patch with two minor tweaks to v005:
          (1) increased randomness in the test case
          (2) reducing the synchronization on init since that method is called in each invokation of RpcInvoker.call. Now it only requires synchronization if a new protocol is added to the cache; this stays more in line with the previous behavior of MutableRates.

          Show
          xkrogen Erik Krogen added a comment - Attaching v006 patch with two minor tweaks to v005: (1) increased randomness in the test case (2) reducing the synchronization on init since that method is called in each invokation of RpcInvoker.call . Now it only requires synchronization if a new protocol is added to the cache; this stays more in line with the previous behavior of MutableRates .
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik! +1 on v6 patch pending Jenkins.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik! +1 on v6 patch pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 41s trunk passed
          +1 compile 10m 49s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 22s trunk passed
          +1 findbugs 1m 30s trunk passed
          +1 javadoc 0m 53s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 9m 18s the patch passed
          +1 javac 9m 18s the patch passed
          -0 checkstyle 0m 33s hadoop-common-project/hadoop-common: The patch generated 2 new + 21 unchanged - 2 fixed = 23 total (was 23)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 37s the patch passed
          +1 javadoc 0m 52s the patch passed
          +1 unit 8m 3s hadoop-common in the patch passed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          47m 30s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13782
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838071/HADOOP-13782.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f440f8122d25 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 29e3b34
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11034/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11034/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11034/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 41s trunk passed +1 compile 10m 49s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 22s trunk passed +1 findbugs 1m 30s trunk passed +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 9m 18s the patch passed +1 javac 9m 18s the patch passed -0 checkstyle 0m 33s hadoop-common-project/hadoop-common: The patch generated 2 new + 21 unchanged - 2 fixed = 23 total (was 23) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 37s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 8m 3s hadoop-common in the patch passed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 47m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13782 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838071/HADOOP-13782.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f440f8122d25 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 29e3b34 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11034/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11034/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11034/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          I just committed the patch to trunk~branch-2.7. Thanks Erik for the contribution!

          Show
          zhz Zhe Zhang added a comment - I just committed the patch to trunk~branch-2.7. Thanks Erik for the contribution!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10793 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10793/)
          HADOOP-13782. Make MutableRates metrics thread-local write, (zhz: rev 77c13c385774c51766fe505397fa916754ac08d4)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcDetailedMetrics.java
          • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRates.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10793 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10793/ ) HADOOP-13782 . Make MutableRates metrics thread-local write, (zhz: rev 77c13c385774c51766fe505397fa916754ac08d4) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcDetailedMetrics.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRates.java

            People

            • Assignee:
              xkrogen Erik Krogen
              Reporter:
              xkrogen Erik Krogen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development