[TEZ-2605] [Umbrella] CPU optimizations for hotspots

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Tsuyoshi Ozawa
• Created: 08/Jul/15 15:13
• Updated: 29/May/16 03:18

Description

There are some CPU hotpots when processing large data when IO is highly optimized:

1. Sort : memory access to compare with 2 values can be bottleneck.
2. Aggregation : hash construction for UnorderedPartitionedKVWriter can be bottleneck.
3. Filter : memory access to compare the key values with given condition.

This issue is a umbrella jira for CPU optmizations at Tez side.

Related works:
Alphasort: http://dl.acm.org/citation.cfm?id=615237
Multi-Core, Main-Memory Joins: Sort vs. Hash Revisited: https://cs.uwaterloo.ca/~tozsu/publications/other/p168-balkesen.pdf

Issue Links

• is related to: HADOOP-11029 FileSystem#Statistics uses volatile variables that must be updated on write or read calls.
• is related to: HADOOP-10694 Remove synchronized input streams from Writable deserialization
• is related to: TEZ-3284 Synchronization for every write in UnorderdKVWriter
• is related to: TEZ-1491 Tez reducer-side merge's counter update is slow
• is related to: TEZ-1277 Tez Spill handler should truncate files to reserve space on disk
• is related to: TEZ-2582 Consider removing DataInputBuffer sync overheads in pipelinedsorter

Sub-Tasks

• Cache-friendly data structure for sorting: Resolved
• SIMD-based bitonic merge sorting: Open
• Fix Tez GenericCounter performance bottleneck: Closed
• Using Unsafe API for swapping kvmeta: Open
• Implement Specific Workaround for JDK-8026049 & JDK-8073093: Closed
• Fix the InputBuffer bottleneck during the in-memory spill merge: Resolved
• Tez: PipelinedSorter - remove interface calls from the inner loop: Open