Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 1.0.12, 1.1.6
    • Component/s: None
    • Labels:
      None

      Description

      I am consistently observing slow-downs in StorageProxy caused by the NonBlockingHashMap used indirectly by MessagingService via the callbacks ExpiringMap.

      This seems do be due to NBHM having unbounded memory usage in the face of workloads with high key churn. As monotonically increasing integers are used as callback id's by MessagingService, the backing NBHM eventually ends up growing the backing store unboundedly. This causes it to also do very large and expensive backing store reallocation and migrations, causing throughput to drop to tens of operations per second, lasting seconds or even minutes.

      This behavior is especially noticable for high throughput workloads where the dataset is completely in ram and I'm doing up to a hundred thousand reads per second.

      Replacing NBHM in ExpiringMap with the java standard library ConcurrentHashMap resolved the issue and allowed me to keep a consistent high throughput.

      An open issue on NBHM can be seen here: http://sourceforge.net/tracker/?func=detail&aid=3563980&group_id=194172&atid=948362

        Activity

        Hide
        danielnorberg Daniel Norberg added a comment -

        Attached patch with the proposed fix; replaces NBHM with CHM in ExpiringMap.

        Show
        danielnorberg Daniel Norberg added a comment - Attached patch with the proposed fix; replaces NBHM with CHM in ExpiringMap.
        Hide
        jbellis Jonathan Ellis added a comment -

        committed, thanks!

        (also commented on NBHM SF issue.)

        Show
        jbellis Jonathan Ellis added a comment - committed, thanks! (also commented on NBHM SF issue.)
        Hide
        jbellis Jonathan Ellis added a comment -

        did a quick audit of NBHM usage elsewhere. think we're okay, everything else has a pretty tiny amount of possible keys. the main "large" maps are in the column containers where NBHM is a non-candidate since we need sorting.

        Show
        jbellis Jonathan Ellis added a comment - did a quick audit of NBHM usage elsewhere. think we're okay, everything else has a pretty tiny amount of possible keys. the main "large" maps are in the column containers where NBHM is a non-candidate since we need sorting.
        Hide
        jbellis Jonathan Ellis added a comment -

        also committed to 1.0 branch

        Show
        jbellis Jonathan Ellis added a comment - also committed to 1.0 branch
        Hide
        scode Peter Schuller added a comment -

        Nice catch!

        FWIW, ConcurrentSkipListMap should probably be considered to avoid locking (though I believe it's generally slower and it'll be less memory compact).

        Show
        scode Peter Schuller added a comment - Nice catch! FWIW, ConcurrentSkipListMap should probably be considered to avoid locking (though I believe it's generally slower and it'll be less memory compact).

          People

          • Assignee:
            danielnorberg Daniel Norberg
            Reporter:
            danielnorberg Daniel Norberg
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development