Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-112

Investigate partitioning in memory map by locality group

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: tserver
    • Labels:

      Description

      Currently the in memory map is not partitioned by locality group. This could negatively impact scan and minor compaction performance. Would like to run some experiments to understand the performance implications. Partitioning by locality group could negatively impact insert performance, it could go from O(log(R)+log(C)) to O(L * (log(R)+log(C))) in the worst case. L is the number of locality groups, R is the number of rows and C is the number of columns. The worst case is where each mutation has a change for each locality group.

      Currently the in memory map is a map of maps. Like the following.

        map<row, map<col, val>>
      

      Could conceptually change this to one of the following. The first is best for scans, that access some locality groups, and minor compactions. The second is good for inserts where the mutation covers all locality groups, because the row is only looked up once.

        map<localityGroup, map<row, map<col, val>>>
      
        map<row, map<localityGroup, map<col, val>>>
      

      The Accumulo native map is implemented using C++,STL, JNI, and with thread locking in java.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kturner Keith Turner
                Reporter:
                kturner Keith Turner
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: