[OAK-9707] Don't fail oak-run indexing on invalid data

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.44.0
• Component/s: indexing, oak-run
• Assignee: Thomas Mueller
• Reporter: Thomas Mueller
• Created: 24/Feb/22 15:15
• Updated: 15/Jul/22 14:53
• Resolved: 28/Mar/22 04:29

Description

Error like the one below currently mean oak-run indexing will fail. Instead, a warning should be logged, and just this field should be removed from the document.

java.lang.IllegalArgumentException: DocValuesField ":dvjcr:content/metadata/dc:title" is too large, must be <= 32766
	at org.apache.lucene.index.SortedDocValuesWriter.addValue(SortedDocValuesWriter.java:68)
	at org.apache.lucene.index.DocValuesProcessor.addSortedField(DocValuesProcessor.java:125)
	at org.apache.lucene.index.DocValuesProcessor.addField(DocValuesProcessor.java:59)
	at org.apache.lucene.index.TwoStoredFieldsConsumers.addField(TwoStoredFieldsConsumers.java:36)
	at org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:236)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:253)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:455)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1534)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1204)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1185)
	at org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.updateDocument(DefaultIndexWriter.java:93)
	at org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.updateDocument(DefaultIndexWriter.java:54)
	at org.apache.jackrabbit.oak.plugins.index.lucene.writer.MultiplexingIndexWriter.updateDocument(MultiplexingIndexWriter.java:60)
	at org.apache.jackrabbit.oak.plugins.index.lucene.writer.MultiplexingIndexWriter.updateDocument(MultiplexingIndexWriter.java:37)
	at org.apache.jackrabbit.oak.index.indexer.document.LuceneIndexer.writeToIndex(LuceneIndexer.java:108)
	at org.apache.jackrabbit.oak.index.indexer.document.LuceneIndexer.index(LuceneIndexer.java:80)
	at org.apache.jackrabbit.oak.index.indexer.document.CompositeIndexer.index(CompositeIndexer.java:58)
	at org.apache.jackrabbit.oak.index.indexer.document.DocumentStoreIndexerBase.reindex(DocumentStoreIndexerBase.java:223)

Issue Links

• is related to: OAK-9731 If property length is > 32766, we should truncate property and truncated property should be searchable
• links to: GitHub Pull Request #522
• links to: GitHub Pull Request #523