Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-6339

MapRecord#getKeys should should initialize child iterables lazily

    XMLWordPrintableJSON

    Details

      Description

      Recently we saw OutOfMemory using oakRepoStats script with a SegmentNodeStore setup where uuid index has 16M+ entries and thus creating a very flat hierarchy. This happened while computing Tree#getChildren iterator which internally invokes MapRecord#getKeys to obtain an iterable for child node names.

      This happened because code in getKeys computes the key list eagerly by calling bucket.getKeys() which recursivly calls same for each child bucket and thus resulting in eager evaluation.

              if (isBranch(size, level)) {
                  List<MapRecord> buckets = getBucketList(segment);
                  List<Iterable<String>> keys =
                          newArrayListWithCapacity(buckets.size());
                  for (MapRecord bucket : buckets) {
                      keys.add(bucket.getKeys());
                  }
                  return concat(keys);
              }
      

      Instead here we should use same approach as used in MapRecord#getEntries i.e. evalate the iterable for child buckets lazily

              if (isBranch(size, level)) {
                  List<MapRecord> buckets = getBucketList(segment);
                  List<Iterable<MapEntry>> entries =
                          newArrayListWithCapacity(buckets.size());
                  for (final MapRecord bucket : buckets) {
                      entries.add(new Iterable<MapEntry>() {
                          @Override
                          public Iterator<MapEntry> iterator() {
                              return bucket.getEntries(diffKey, diffValue).iterator();
                          }
                      });
                  }
                  return concat(entries);
              }
      

        Attachments

        1. OAK-6339-1.6.patch
          1 kB
          Michael Dürig

          Activity

            People

            • Assignee:
              mduerig Michael Dürig
              Reporter:
              chetanm Chetan Mehrotra
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: