[OAK-7103] Enable compression by default on DocumentStoreIndexer logic

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.8.0
• Component/s: run
• Assignee: Chetan Mehrotra
• Reporter: Chetan Mehrotra
• Created: 21/Dec/17 04:13
• Updated: 12/Jan/18 11:58
• Resolved: 21/Dec/17 04:47

Description

While performing tests it appears that enabling end to end compression reduces the sorting time by 14 mins (39.87 min to 26.44 min) and disk consumption by 65GB (87GB to 12.5). Based on that we should enable compression by default for

1. Create compressed base store.json written by traversal
2. Enable compression for intermediate files created while sorting
3. Enable compression for finally sorted json file

Issue Links

• is blocked by: OAK-7104 Support read and write to compressed file in ExternalSort