[OAK-5859] Analyse and reduce IO amplification by OS

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: segment-tar
• Labels: performance scalability
• Epic Link: TarMK content scalability
• Assignee: Unassigned
• Reporter: Michael Dürig
• Created: 28/Feb/17 09:39
• Updated: 06/Feb/18 21:31
• Resolved: 06/Feb/18 21:31

Description

Certain operation system settings might result in too much data actually being read from disk causing early setting on of thrashing. E.g. transparent huge pages or too big read aheads might be contra productive in combination with the TarMKs memory mapping model.

• Determine the ratio of data being read by the TarMK and actual data being read from disk. Determine the impact of relevant OS parameters (e.g. transparent huge pages) on this ratio.

• Compare memory mapped mode with file IO mode with an accordingly increased segment cache. This would move prediction of what is likely to be read next from the OS layer into our segment cache eviction strategy.

Issue Links

• depends upon: OAK-5634 Expose IOMonitor stats via JMX
• is depended upon by: OAK-5860 Compressed segments
• relates to: OAK-5786 Memory based FileStore bootstrap