Accumulo
  1. Accumulo
  2. ACCUMULO-2936 Running Accumulo on different JVMs
  3. ACCUMULO-2942

org.apache.accumulo.core.util.format.ShardedTableDistributionFormatterTest.testAggregate failure

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.6.1, 1.7.0
    • Component/s: tserver
    • Labels:
    • Environment:

      IBM JVM

      Description

      org.apache.accumulo.core.util.format.ShardedTableDistributionFormatterTest.testAggregate . This fails on IBM JRE, since the test is asserting order of elements in a HashMap. This consistently passes on Sun , and consistently fails on Oracle.

      The ShardedTableDistributionFormatter inherits from AggregatingFormatter which has 2 overriding methods - aggregateStats and getStats. In the ShardedTableDistributionFormatter implementation, the aggregateStats prepares a list based on the HashMap, and the getStats creates a string by serializing values in the HashMap.

      Due to the unpredictability of Hash ordering in different Java versions (even same vendor, different versions), the getStats() output is inconsistent. This is not a problem in itself. However since we are asserting on the content of getStats, we we either make the getStatus consistent or we do some refactoring and do 2 tests - one test on the structure that getStats is serializing, and another test to assert the output of getStats based on a predictable structure.

      Some people expressed concern for changing the underlying structure from a HashMap to TreeMap due to performance considerations. Question is, is this code ever executed in such an environment to be concerned about this?

      Alternatively, we could just change the getStats method, which is after the 'heavy-lifting' of iterating over all entries. The stats that are calculated are aggregates per day. Therefore this will not be such a large structure, and could then be sorted before being output.

        Issue Links

          Activity

          Hide
          Sean Busbey added a comment -

          please update the test to account for the lack of ordering.

          Show
          Sean Busbey added a comment - please update the test to account for the lack of ordering.
          Hide
          ASF GitHub Bot added a comment -

          GitHub user haydenmarchant opened a pull request:

          https://github.com/apache/accumulo/pull/10

          ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs

          ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs

          ShardedTableDistributionFormatterTest.testAggregate() was failing due
          to inconsistency in order to stats that were returned in the getStats()
          method. This is due to the fact that the structure that holds the stats is
          a HashMap.

          The test was changed so that it does not assert that elements exists
          at certain location in results, but rather it now checks that elements
          exist in the list. The test was also slightly broadened so that it
          checks that tuples of (day, count) exist in list.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/haydenmarchant/accumulo ACCUMULO-2942

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/accumulo/pull/10.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #10


          commit 72a95412c756463bce99e8610377a018a28e8a7d
          Author: haydenmarchant <hayden.marchant@gmail.com>
          Date: 2014-06-25T13:19:31Z

          ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs

          ShardedTableDistributionFormatterTest.testAggregate() was failing due
          to inconsistency in order to stats that were returned in the getStats()
          method. This is due to the fact that the structure that holds the stats is
          a HashMap.

          The test was changed so that it does not assert that elements exists
          at certain location in results, but rather it now checks that elements
          exist in the list. The test was also slightly broadened so that it
          checks that tuples of (day, count) exist in list.

          commit af70970ef5602a6f489f7fe2259dffca0edbd901
          Author: haydenmarchant <hayden.marchant@gmail.com>
          Date: 2014-06-25T13:34:12Z

          ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs

          ShardedTableDistributionFormatterTest.testAggregate() was failing due
          to inconsistency in order to stats that were returned in the getStats()
          method. This is due to the fact that the structure that holds the stats is
          a HashMap.

          The test was changed so that it does not assert that elements exists
          at certain location in results, but rather it now checks that elements
          exist in the list. The test was also slightly broadened so that it
          checks that tuples of (day, count) exist in list.


          Show
          ASF GitHub Bot added a comment - GitHub user haydenmarchant opened a pull request: https://github.com/apache/accumulo/pull/10 ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs ShardedTableDistributionFormatterTest.testAggregate() was failing due to inconsistency in order to stats that were returned in the getStats() method. This is due to the fact that the structure that holds the stats is a HashMap. The test was changed so that it does not assert that elements exists at certain location in results, but rather it now checks that elements exist in the list. The test was also slightly broadened so that it checks that tuples of (day, count) exist in list. You can merge this pull request into a Git repository by running: $ git pull https://github.com/haydenmarchant/accumulo ACCUMULO-2942 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/accumulo/pull/10.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10 commit 72a95412c756463bce99e8610377a018a28e8a7d Author: haydenmarchant <hayden.marchant@gmail.com> Date: 2014-06-25T13:19:31Z ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs ShardedTableDistributionFormatterTest.testAggregate() was failing due to inconsistency in order to stats that were returned in the getStats() method. This is due to the fact that the structure that holds the stats is a HashMap. The test was changed so that it does not assert that elements exists at certain location in results, but rather it now checks that elements exist in the list. The test was also slightly broadened so that it checks that tuples of (day, count) exist in list. commit af70970ef5602a6f489f7fe2259dffca0edbd901 Author: haydenmarchant <hayden.marchant@gmail.com> Date: 2014-06-25T13:34:12Z ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs ShardedTableDistributionFormatterTest.testAggregate() was failing due to inconsistency in order to stats that were returned in the getStats() method. This is due to the fact that the structure that holds the stats is a HashMap. The test was changed so that it does not assert that elements exists at certain location in results, but rather it now checks that elements exist in the list. The test was also slightly broadened so that it checks that tuples of (day, count) exist in list.
          Hide
          ASF GitHub Bot added a comment -

          Github user busbey commented on the pull request:

          https://github.com/apache/accumulo/pull/10#issuecomment-47117689

          please squash commits

          Show
          ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47117689 please squash commits
          Hide
          ASF GitHub Bot added a comment -

          Github user busbey commented on the pull request:

          https://github.com/apache/accumulo/pull/10#issuecomment-47118043

          Please format the files according to the [style guide][1] (notably you aren't consistently using 2 spaces per indent level)

          [1]: http://accumulo.apache.org/source.html#coding-practices

          Show
          ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47118043 Please format the files according to the [style guide] [1] (notably you aren't consistently using 2 spaces per indent level) [1] : http://accumulo.apache.org/source.html#coding-practices
          Hide
          ASF GitHub Bot added a comment -

          Github user busbey commented on the pull request:

          https://github.com/apache/accumulo/pull/10#issuecomment-47118595

          I've assigned this to you over on the jira for ACCUMULO-2942[1]. You're listed as a contributor in jira now, which should allow you to adopt tickets on your own.

          When working on things, please assign the jira to yourself. When you open a PR, please change the jira status to "Patch Submitted".

          [1]: https://issues.apache.org/jira/browse/ACCUMULO-2942

          Show
          ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47118595 I've assigned this to you over on the jira for ACCUMULO-2942 [1] . You're listed as a contributor in jira now, which should allow you to adopt tickets on your own. When working on things, please assign the jira to yourself. When you open a PR, please change the jira status to "Patch Submitted". [1] : https://issues.apache.org/jira/browse/ACCUMULO-2942
          Hide
          ASF GitHub Bot added a comment -

          Github user haydenmarchant commented on the pull request:

          https://github.com/apache/accumulo/pull/10#issuecomment-47521353

          formatting has been fixed.

          Show
          ASF GitHub Bot added a comment - Github user haydenmarchant commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47521353 formatting has been fixed.
          Hide
          ASF GitHub Bot added a comment -

          Github user busbey commented on the pull request:

          https://github.com/apache/accumulo/pull/10#issuecomment-47538043

          Looks good. I'm in the process of rebasing to the current branch and pushing.

          Show
          ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47538043 Looks good. I'm in the process of rebasing to the current branch and pushing.
          Hide
          Sean Busbey added a comment -

          Thanks for the patch Hayden!

          Would you like to be listed on our contributors page?

          If so, what would you like your employer listed as? What timezone?

          Show
          Sean Busbey added a comment - Thanks for the patch Hayden! Would you like to be listed on our contributors page ? If so, what would you like your employer listed as? What timezone?
          Hide
          ASF GitHub Bot added a comment -

          Github user busbey commented on the pull request:

          https://github.com/apache/accumulo/pull/10#issuecomment-47563841

          this has been pushed and merged forward.

          Show
          ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47563841 this has been pushed and merged forward.
          Hide
          ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/accumulo/pull/10

          Show
          ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/accumulo/pull/10

            People

            • Assignee:
              Hayden Marchant
              Reporter:
              Hayden Marchant
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Time Spent - 20m Remaining Estimate - 23h 40m
                23h 40m
                Logged:
                Time Spent - 20m Remaining Estimate - 23h 40m
                20m

                  Development