[SPARK-41318] Validate scalability improvements due to JDK-8180450 on Netty 4.1.84.Final

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Spark Core
• Labels: jdk netty performance scalability
• Assignee: Unassigned
• Reporter: Francesco Nigro
• Created: 29/Nov/22 09:14
• Updated: 29/Nov/22 09:16

Description

https://netty.io/news/2022/10/11/4-1-84-Final.html has introduced a bug-fix/improvement to prevent a JDK scalability issue due to type check against interfaces (see https://github.com/netty/netty/pull/12806 for more details - and https://redhatperf.github.io/post/type-check-scalability-issue/ for a deeper and wider analysis).

it seems that Spark network make uses of Netty channel duplex handlers on TransportContext and it's likely to be affected by it (or better, depending which netty version).

Moreover, I suggest to verify if there are other places where the issue can happen and eventually, report them on https://bugs.openjdk.org/browse/JDK-8180450 as well (if they use type checks in an idiomatic form).
The blog post has linked https://github.com/RedHatPerf/type-pollution-agent to detect dangerous candidate code paths.

Issue Links

• is fixed by: SPARK-40794 Upgrade Netty from 4.1.80 to 4.1.84