[YARN-7037] Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.3
• Component/s: nodemanager
• Target Version/s: 2.8.2
• Hadoop Flags: Reviewed
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 17/Aug/17 11:24
• Updated: 25/Oct/19 20:26
• Resolved: 29/Aug/17 22:23

Description

Split this improvement from YARN-6259.
It's useful to read container logs more efficiently. With zero-copy approach, data transfer pipeline (disk --> read buffer --> NM buffer --> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket buffer) .
In my local test, time cost of copying 256MB file with zero-copy can be reduced from 12 seconds to 2.5 seconds.