Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19552

Upgrade Netty version to 4.1.x final

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.3.0
    • Component/s: Build
    • Labels:
      None
    • Flags:
      Important

      Description

      Netty 4.1.8 was recently released but isn't API compatible with previous major versions (like Netty 4.0.x), see http://netty.io/news/2017/01/30/4-0-44-Final-4-1-8-Final.html for details.

      This version does include a fix for a security concern but not one we'd be exposed to with Spark "out of the box". Let's upgrade the version we use to be on the safe side as the security fix I'm especially interested in is not available in the 4.0.x release line.

      We should move up anyway to take on a bunch of other big fixes cited in the release notes (and if anyone were to use Spark with netty and tcnative, they shouldn't be exposed to the security problem) - we should be good citizens and make this change.

      As this 4.1 version involves API changes we'll need to implement a few methods and possibly adjust the Sasl tests. This JIRA and associated pull request starts the process which I'll work on - and any help would be much appreciated! Currently I know:

      @Override
      public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise)
            throws Exception {
            if (!foundEncryptionHandler) {
              foundEncryptionHandler =
                ctx.channel().pipeline().get(encryptHandlerName) != null; <-- this returns false and causes test failures
            }
            ctx.write(msg, promise);
          }
      

      Here's what changes will be required (at least):

      common/network-common/src/main/java/org/apache/spark/network/crypto/TransportCipher.java

      requires touch, retain and transferred methods

      common/network-common/src/main/java/org/apache/spark/network/sasl/SaslEncryption.java

      requires the above methods too

      common/network-common/src/test/java/org/apache/spark/network/protocol/MessageWithHeaderSuite.java

      With "dummy" implementations so we can at least compile and test, we'll see five new test failures to address.

      These are

      org.apache.spark.network.sasl.SparkSaslSuite.testFileRegionEncryption
      org.apache.spark.network.sasl.SparkSaslSuite.testSaslEncryption
      org.apache.spark.network.shuffle.ExternalShuffleSecuritySuite.testEncryption
      org.apache.spark.rpc.netty.NettyRpcEnvSuite.send with SASL encryption
      org.apache.spark.rpc.netty.NettyRpcEnvSuite.ask with SASL encryption
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bryanc Bryan Cutler
                Reporter:
                aroberts Adam Roberts
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: