[OAK-9138] Have a mechanism to track failed docs in ES

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.34.0
• Assignee: Nitin Gupta
• Reporter: Nitin Gupta
• Created: 10/Jul/20 08:39
• Updated: 10/Sep/20 15:45
• Resolved: 15/Jul/20 11:45

Description

Currently we send bulk request with multiple docs to be indexed in ES.

There could be cases where a request could fail completely or partially (but this is async) and the information whether the bulk req passed or failed or passed but had some docs that got failed to be indexed is available via callbacks of the bulk processor listener ([0]).

 

The method at [1] returns true if even 1 of the bulk processor updates worked for this particular processor instance (this is as per the contract) - while this made sense for lucene (where we were sending 1 req at a time), with ES this might not make much sense.

 

So we need a mechanism to track the docs that got failed to get indexed on ES and accordingly set the state of the index.

 

[0]: https://github.com/apache/jackrabbit-oak/blob/trunk/oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elasticsearch/index/ElasticsearchIndexWriter.java#L210#L239

[1]: https://github.com/apache/jackrabbit-oak/blob/trunk/oak-search-elastic/src/main/java/org/apache/jackrabbit/oak/plugins/index/elasticsearch/index/ElasticsearchIndexWriter.java#L109