[OAK-6339] MapRecord#getKeys should should initialize child iterables lazily

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.7.3, 1.8.0
• Component/s: segment-tar
• Assignee: Michael Dürig
• Reporter: Chetan Mehrotra
• Created: 13/Jun/17 10:19
• Updated: 31/Jul/22 10:36
• Resolved: 14/Jun/17 13:15

Description

Recently we saw OutOfMemory using oakRepoStats script with a SegmentNodeStore setup where uuid index has 16M+ entries and thus creating a very flat hierarchy. This happened while computing Tree#getChildren iterator which internally invokes MapRecord#getKeys to obtain an iterable for child node names.

This happened because code in getKeys computes the key list eagerly by calling bucket.getKeys() which recursivly calls same for each child bucket and thus resulting in eager evaluation.

        if (isBranch(size, level)) {
            List<MapRecord> buckets = getBucketList(segment);
            List<Iterable<String>> keys =
                    newArrayListWithCapacity(buckets.size());
            for (MapRecord bucket : buckets) {
                keys.add(bucket.getKeys());
            }
            return concat(keys);
        }

Instead here we should use same approach as used in MapRecord#getEntries i.e. evalate the iterable for child buckets lazily

        if (isBranch(size, level)) {
            List<MapRecord> buckets = getBucketList(segment);
            List<Iterable<MapEntry>> entries =
                    newArrayListWithCapacity(buckets.size());
            for (final MapRecord bucket : buckets) {
                entries.add(new Iterable<MapEntry>() {
                    @Override
                    public Iterator<MapEntry> iterator() {
                        return bucket.getEntries(diffKey, diffValue).iterator();
                    }
                });
            }
            return concat(entries);
        }