[TEZ-4157] ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from tez

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.1, 0.9.3
• Labels: ShuffleHandler
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 27/Apr/20 11:32
• Updated: 09/Dec/22 09:39
• Resolved: 23/Apr/21 10:12

Description

In the dependency tree, there are 2 occurrences of compile scope direct netty dependencies, however, they're not used at all. I compiled locally successfully without them. E.g. when investigating blackduck alerts (complaining about netty deps for current 3.10.5.Final), it would be cleaner to start from a dependency tree where Tez doesn't depend on netty directly in order to eliminate its responsibility (and move the focus to underlying hadoop for instance).

Tez depends on netty3 almost only in ShuffleHandler and some related classes. We can eliminate netty3 by upgrading it, but this effort might involve some testing due to fundamental changes from netty3->netty4 + we don't have a reference yet, as hadoop's ShuffleHandler is still on netty3.

As per the netty documentation, we can also expect some performance improvement (e.g. Pooled buffers).

Background:
netty4 migration guideline: https://netty.io/wiki/new-and-noteworthy-in-4.0.html
articles of possible performance improvement:
https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html
https://developer.squareup.com/blog/upgrading-a-reverse-proxy-from-netty-3-to-4/

Issue Links

• relates to: TEZ-4394 Netty4 ShuffleHandler: should use 1 boss thread
• relates to: HIVE-24524 LLAP ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from hive where it's possible
• links to: GitHub Pull Request #118