Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-6339

MapRecord#getKeys should should initialize child iterables lazily

    XMLWordPrintableJSON

Details

    Description

      Recently we saw OutOfMemory using oakRepoStats script with a SegmentNodeStore setup where uuid index has 16M+ entries and thus creating a very flat hierarchy. This happened while computing Tree#getChildren iterator which internally invokes MapRecord#getKeys to obtain an iterable for child node names.

      This happened because code in getKeys computes the key list eagerly by calling bucket.getKeys() which recursivly calls same for each child bucket and thus resulting in eager evaluation.

              if (isBranch(size, level)) {
                  List<MapRecord> buckets = getBucketList(segment);
                  List<Iterable<String>> keys =
                          newArrayListWithCapacity(buckets.size());
                  for (MapRecord bucket : buckets) {
                      keys.add(bucket.getKeys());
                  }
                  return concat(keys);
              }
      

      Instead here we should use same approach as used in MapRecord#getEntries i.e. evalate the iterable for child buckets lazily

              if (isBranch(size, level)) {
                  List<MapRecord> buckets = getBucketList(segment);
                  List<Iterable<MapEntry>> entries =
                          newArrayListWithCapacity(buckets.size());
                  for (final MapRecord bucket : buckets) {
                      entries.add(new Iterable<MapEntry>() {
                          @Override
                          public Iterator<MapEntry> iterator() {
                              return bucket.getEntries(diffKey, diffValue).iterator();
                          }
                      });
                  }
                  return concat(entries);
              }
      

      Attachments

        1. OAK-6339-1.6.patch
          1 kB
          Michael Dürig

        Activity

          People

            mduerig Michael Dürig
            chetanm Chetan Mehrotra
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: