[JAMES-2555] Re-indexing big mailboxes fails

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, master, 3.0.1, 3.1.0
• Fix Version/s: 3.2.0
• Component/s: CLI, mailbox
• Labels: perf
• Assignee: Unassigned
• Reporter: Benoit Tellier
• Created: 09/Oct/18 02:36
• Updated: 05/Dec/18 09:51
• Resolved: 23/Oct/18 01:48

Description

Reported internally:

Doing a reindex via the CLI on a mailbox which contains ~24000 mails does not work.

 -  heap out of memory issue

Heap increased, then:

 -  cassandra timeout when increasing heap

Timeout increased, then:

 -  heap out of memory issue

Heap increased, then:

 -  GC overhead limit exceeded

In ReIndexerImpl instead of doing a FULL read to reindex the all mailbox (AND all of the related data) we should rather do a first metadata query to load uids then perform subsequent single reads.

That is the root issue: trying to load 11 Go of mails in memory.

Furthermore, a simple error should not trigger the end of a ReIndexing.