Accumulo
  1. Accumulo
  2. ACCUMULO-2936 Running Accumulo on different JVMs
  3. ACCUMULO-2942

org.apache.accumulo.core.util.format.ShardedTableDistributionFormatterTest.testAggregate failure

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.6.1, 1.7.0
    • Component/s: tserver
    • Labels:
    • Environment:

      IBM JVM

      Description

      org.apache.accumulo.core.util.format.ShardedTableDistributionFormatterTest.testAggregate . This fails on IBM JRE, since the test is asserting order of elements in a HashMap. This consistently passes on Sun , and consistently fails on Oracle.

      The ShardedTableDistributionFormatter inherits from AggregatingFormatter which has 2 overriding methods - aggregateStats and getStats. In the ShardedTableDistributionFormatter implementation, the aggregateStats prepares a list based on the HashMap, and the getStats creates a string by serializing values in the HashMap.

      Due to the unpredictability of Hash ordering in different Java versions (even same vendor, different versions), the getStats() output is inconsistent. This is not a problem in itself. However since we are asserting on the content of getStats, we we either make the getStatus consistent or we do some refactoring and do 2 tests - one test on the structure that getStats is serializing, and another test to assert the output of getStats based on a predictable structure.

      Some people expressed concern for changing the underlying structure from a HashMap to TreeMap due to performance considerations. Question is, is this code ever executed in such an environment to be concerned about this?

      Alternatively, we could just change the getStats method, which is after the 'heavy-lifting' of iterating over all entries. The stats that are calculated are aggregates per day. Therefore this will not be such a large structure, and could then be sorted before being output.

        Activity

        Hide
        Sean Busbey added a comment -

        please update the test to account for the lack of ordering.

        Show
        Sean Busbey added a comment - please update the test to account for the lack of ordering.
        Hide
        ASF GitHub Bot added a comment -

        GitHub user haydenmarchant opened a pull request:

        https://github.com/apache/accumulo/pull/10

        ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs

        ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs

        ShardedTableDistributionFormatterTest.testAggregate() was failing due
        to inconsistency in order to stats that were returned in the getStats()
        method. This is due to the fact that the structure that holds the stats is
        a HashMap.

        The test was changed so that it does not assert that elements exists
        at certain location in results, but rather it now checks that elements
        exist in the list. The test was also slightly broadened so that it
        checks that tuples of (day, count) exist in list.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/haydenmarchant/accumulo ACCUMULO-2942

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/accumulo/pull/10.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #10


        commit 72a95412c756463bce99e8610377a018a28e8a7d
        Author: haydenmarchant <hayden.marchant@gmail.com>
        Date: 2014-06-25T13:19:31Z

        ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs

        ShardedTableDistributionFormatterTest.testAggregate() was failing due
        to inconsistency in order to stats that were returned in the getStats()
        method. This is due to the fact that the structure that holds the stats is
        a HashMap.

        The test was changed so that it does not assert that elements exists
        at certain location in results, but rather it now checks that elements
        exist in the list. The test was also slightly broadened so that it
        checks that tuples of (day, count) exist in list.

        commit af70970ef5602a6f489f7fe2259dffca0edbd901
        Author: haydenmarchant <hayden.marchant@gmail.com>
        Date: 2014-06-25T13:34:12Z

        ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs

        ShardedTableDistributionFormatterTest.testAggregate() was failing due
        to inconsistency in order to stats that were returned in the getStats()
        method. This is due to the fact that the structure that holds the stats is
        a HashMap.

        The test was changed so that it does not assert that elements exists
        at certain location in results, but rather it now checks that elements
        exist in the list. The test was also slightly broadened so that it
        checks that tuples of (day, count) exist in list.


        Show
        ASF GitHub Bot added a comment - GitHub user haydenmarchant opened a pull request: https://github.com/apache/accumulo/pull/10 ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs ShardedTableDistributionFormatterTest.testAggregate() was failing due to inconsistency in order to stats that were returned in the getStats() method. This is due to the fact that the structure that holds the stats is a HashMap. The test was changed so that it does not assert that elements exists at certain location in results, but rather it now checks that elements exist in the list. The test was also slightly broadened so that it checks that tuples of (day, count) exist in list. You can merge this pull request into a Git repository by running: $ git pull https://github.com/haydenmarchant/accumulo ACCUMULO-2942 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/accumulo/pull/10.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10 commit 72a95412c756463bce99e8610377a018a28e8a7d Author: haydenmarchant <hayden.marchant@gmail.com> Date: 2014-06-25T13:19:31Z ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs ShardedTableDistributionFormatterTest.testAggregate() was failing due to inconsistency in order to stats that were returned in the getStats() method. This is due to the fact that the structure that holds the stats is a HashMap. The test was changed so that it does not assert that elements exists at certain location in results, but rather it now checks that elements exist in the list. The test was also slightly broadened so that it checks that tuples of (day, count) exist in list. commit af70970ef5602a6f489f7fe2259dffca0edbd901 Author: haydenmarchant <hayden.marchant@gmail.com> Date: 2014-06-25T13:34:12Z ACCUMULO-2942 Fixes ShardedTableDistributionFormatterTest on all JVMs ShardedTableDistributionFormatterTest.testAggregate() was failing due to inconsistency in order to stats that were returned in the getStats() method. This is due to the fact that the structure that holds the stats is a HashMap. The test was changed so that it does not assert that elements exists at certain location in results, but rather it now checks that elements exist in the list. The test was also slightly broadened so that it checks that tuples of (day, count) exist in list.
        Hide
        ASF GitHub Bot added a comment -

        Github user busbey commented on the pull request:

        https://github.com/apache/accumulo/pull/10#issuecomment-47117689

        please squash commits

        Show
        ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47117689 please squash commits
        Hide
        ASF GitHub Bot added a comment -

        Github user busbey commented on the pull request:

        https://github.com/apache/accumulo/pull/10#issuecomment-47118043

        Please format the files according to the [style guide][1] (notably you aren't consistently using 2 spaces per indent level)

        [1]: http://accumulo.apache.org/source.html#coding-practices

        Show
        ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47118043 Please format the files according to the [style guide] [1] (notably you aren't consistently using 2 spaces per indent level) [1] : http://accumulo.apache.org/source.html#coding-practices
        Hide
        ASF GitHub Bot added a comment -

        Github user busbey commented on the pull request:

        https://github.com/apache/accumulo/pull/10#issuecomment-47118595

        I've assigned this to you over on the jira for ACCUMULO-2942[1]. You're listed as a contributor in jira now, which should allow you to adopt tickets on your own.

        When working on things, please assign the jira to yourself. When you open a PR, please change the jira status to "Patch Submitted".

        [1]: https://issues.apache.org/jira/browse/ACCUMULO-2942

        Show
        ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47118595 I've assigned this to you over on the jira for ACCUMULO-2942 [1] . You're listed as a contributor in jira now, which should allow you to adopt tickets on your own. When working on things, please assign the jira to yourself. When you open a PR, please change the jira status to "Patch Submitted". [1] : https://issues.apache.org/jira/browse/ACCUMULO-2942
        Hide
        ASF GitHub Bot added a comment -

        Github user haydenmarchant commented on the pull request:

        https://github.com/apache/accumulo/pull/10#issuecomment-47521353

        formatting has been fixed.

        Show
        ASF GitHub Bot added a comment - Github user haydenmarchant commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47521353 formatting has been fixed.
        Hide
        ASF GitHub Bot added a comment -

        Github user busbey commented on the pull request:

        https://github.com/apache/accumulo/pull/10#issuecomment-47538043

        Looks good. I'm in the process of rebasing to the current branch and pushing.

        Show
        ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47538043 Looks good. I'm in the process of rebasing to the current branch and pushing.
        Hide
        Sean Busbey added a comment -

        Thanks for the patch Hayden!

        Would you like to be listed on our contributors page?

        If so, what would you like your employer listed as? What timezone?

        Show
        Sean Busbey added a comment - Thanks for the patch Hayden! Would you like to be listed on our contributors page ? If so, what would you like your employer listed as? What timezone?
        Hide
        ASF GitHub Bot added a comment -

        Github user busbey commented on the pull request:

        https://github.com/apache/accumulo/pull/10#issuecomment-47563841

        this has been pushed and merged forward.

        Show
        ASF GitHub Bot added a comment - Github user busbey commented on the pull request: https://github.com/apache/accumulo/pull/10#issuecomment-47563841 this has been pushed and merged forward.
        Hide
        ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/accumulo/pull/10

        Show
        ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/accumulo/pull/10

          People

          • Assignee:
            Hayden Marchant
            Reporter:
            Hayden Marchant
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Time Spent - 20m Remaining Estimate - 23h 40m
              23h 40m
              Logged:
              Time Spent - 20m Remaining Estimate - 23h 40m
              20m

                Development