[OAK-10048] DocumentStoreIndexerBase#buildFlatFileStore outputs the wrong path when FlatFileSplitter used - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.48.0
Component/s: oak-run
Labels:
None

Description

DocumentStoreIndexerBase#buildFlatFileStore below outputs the first FlatFileStore and its path which is incorrect if the FlatFileSplitter is triggered to split files based on config and index definitions.

Also, the buildFlatFileStoreList method always passes false to split but shoud instead use the configuration IndexerConfiguration.parallelIndexEnabled() to try splitting based on this configuration.

The FlatFileNodeStoreBuilder#buildList needs to handle already available split files as well.

public FlatFileStore buildFlatFileStore() throws IOException, CommitFailedException {
        NodeState checkpointedState = indexerSupport.retrieveNodeStateForCheckpoint();
        Set<String> preferredPathElements = new HashSet<>();
        Set<IndexDefinition> indexDefinitions = getIndexDefinitions();
        for (IndexDefinition indexDf : indexDefinitions) {
            preferredPathElements.addAll(indexDf.getRelativeNodeNames());
        }
        Predicate<String> predicate = s -> indexDefinitions.stream().anyMatch(indexDef -> indexDef.getPathFilter().filter(s) != PathFilter.Result.EXCLUDE);
        FlatFileStore flatFileStore = buildFlatFileStoreList(checkpointedState, null, predicate, preferredPathElements, false, indexDefinitions).get(0);
        log.info("FlatFileStore built at {}. To use this flatFileStore in a reindex step, set System Property-{} with value {}",
                flatFileStore.getFlatFileStorePath(), OAK_INDEXER_SORTED_FILE_PATH, flatFileStore.getFlatFileStorePath());
        return flatFileStore;
}