[IGNITE-8597] Direct data load

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: general
• Labels: iep-22 performance
• Assignee: Unassigned
• Reporter: Vladimir Ozerov
• Created: 24/May/18 09:51
• Updated: 23/May/20 15:01

Description

We should implement optimized data loading mode, which will bypass as much internal Ignite components as possible to improve data loading speed.

Raw design:
1) Direct data load must be performed in exclusive mode, i.e. nobody else are allowed to read or write to specific cache/table at this time. I.e. we need to implement distributed table/cache locks.
2) At first we should write to data pages directly skipping free lists, PK hash index and secondary indexes
3) Once loading is finished, we should rebuild free lists and indexes (bottom-up); external merge implementation will be required
4) We should distinguish between initial load and incremental load. The latter would require merge of indexes with previously available data.

Issue Links

• is related to: IGNITE-11793 Improve isolated updater mode.
• is related to: IGNITE-9314 MVCC TX: Datastreamer operations
• relates to: IGNITE-13063 Bottom-up index rebuild