https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with Cassandra 3.0.x, has some serious bugs around memory handling for SSL connections.
It would be nice if both were updated to 4.0.42, a version released this year.
4.0.23 makes it impossible for me to run SSL, because nodes run out of memory every ~30 minutes. This was fixed in 4.0.27.