[OAK-9573] Make TraverseWithSort strategy the default while indexing

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.42.0
• Component/s: indexing
• Assignee: Amrit Verma
• Reporter: Amrit Verma
• Created: 11/Sep/21 14:34
• Updated: 12/Jan/22 13:44
• Resolved: 16/Sep/21 12:02

Description

With the multithreaded strategy we have seen some issues in synchronization. Eg with thread size 4 -

All 4 running tasks got the MemoryManager's signal, which is indicated by the log lines -

13:54:28.397 [Service Thread] INFO  o.a.j.o.i.i.d.f.TraverseAndSortTask - TAST-NSET19 Setting dumpData to true
13:54:28.398 [Service Thread] INFO  o.a.j.o.i.i.d.f.TraverseAndSortTask - TAST-NSET2 Setting dumpData to true
13:54:28.398 [Service Thread] INFO  o.a.j.o.i.i.d.f.TraverseAndSortTask - TAST-NSET10 Setting dumpData to true
13:54:28.398 [Service Thread] INFO  o.a.j.o.i.i.d.f.TraverseAndSortTask - TAST-NSET1 Setting dumpData to true

However only three of them actually proceeded with dumping the data and notifying the MemoryManager, which is indicated by the log lines -

13:54:33.667 [pool-3-thread-2] INFO  o.a.j.o.i.i.d.f.TraverseAndSortTask - TAST-NSET2 Finished saving data to disk. Notifying memory listener.
13:54:34.588 [pool-3-thread-4] INFO  o.a.j.o.i.i.d.f.TraverseAndSortTask - TAST-NSET19 Finished saving data to disk. Notifying memory listener.
13:54:35.438 [pool-3-thread-3] INFO  o.a.j.o.i.i.d.f.TraverseAndSortTask - TAST-NSET10 Finished saving data to disk. Notifying memory listener.

After this newer tasks weren't able to register and process stuck in some cases.

So we need to make traverse with sort the default strategy for now.

Issue Links

• is related to: OAK-9434 MongoDB indexing: implement parallel chunk download
• links to: GitHub Pull Request #369