[HADOOP-14558] RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.6.0
• Component/s: hdfs-client
• Labels: impala metadata rpc
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 20/Jun/17 23:04
• Updated: 25/Oct/17 19:27
• Resolved: 21/Jun/17 11:25

Description

While running a performance tests for Impala comparing secure and un-secure clusters I noticed that metadata loading operations are 10x slower on a cluster with Kerberos+SSL enabled.
hadoop.rpc.protection is set to privacy

Any recommendations on how this can be mitigated? 10x slowdown is a big hit for metadata loading.

The majority of the slowdown is coming from the two threads below.

Stack Trace	Sample Count	Percentage(%)
org.apache.hadoop.ipc.Client$Connection.run()	5,212	46.586
   org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse()	5,203	46.505
      java.io.DataInputStream.readInt()	5,039	45.039
         java.io.BufferedInputStream.read()	5,038	45.03
            java.io.BufferedInputStream.fill()	5,038	45.03
               org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int)	5,036	45.013
                  java.io.FilterInputStream.read(byte[], int, int)	5,036	45.013
                     org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int, int)	5,036	45.013
                        org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()	5,035	45.004
                           com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int)	4,775	42.68
                              sun.security.jgss.GSSContextImpl.unwrap(byte[], int, int, MessageProp)	4,775	42.68
                                 sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp)	4,768	42.617
                                    sun.security.jgss.krb5.WrapToken.getData()	4,714	42.134
                                       sun.security.jgss.krb5.WrapToken.getData(byte[], int)	4,714	42.134
                                          sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int)	4,714	42.134
                                             sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int, byte[], int)	3,083	27.556
                                                sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int, int, byte[], int)	3,078	27.512
                                                   sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[], byte[], int, int)	3,076	27.494
                                                      sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[], byte[], int, int)	3,076	27.494

And

Stack Trace	Sample Count	Percentage(%)
java.lang.Thread.run()	3,379	30.202
   java.util.concurrent.ThreadPoolExecutor$Worker.run()	3,379	30.202
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)	3,379	30.202
         java.util.concurrent.FutureTask.run()	3,367	30.095
            java.util.concurrent.Executors$RunnableAdapter.call()	3,367	30.095
               org.apache.hadoop.ipc.Client$Connection$3.run()	3,367	30.095
                  java.io.DataOutputStream.flush()	3,367	30.095
                     java.io.BufferedOutputStream.flush()	3,367	30.095
                        java.io.BufferedOutputStream.flushBuffer()	3,367	30.095
                           org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[], int, int)	3,367	30.095
                              com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int)	3,281	29.326
                                 sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp)	3,281	29.326
                                    sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp)	3,280	29.317
                                       sun.security.jgss.krb5.WrapToken.<init>(Krb5Context, MessageProp, byte[], int, int)	1,872	16.732
                                       sun.security.jgss.krb5.WrapToken.encode()	1,407	12.576

This is the Impala Catalog thread which initiates the NameNode request

Stack Trace	Sample Count	Percentage(%)
org.apache.impala.service.JniCatalog.resetMetadata(byte[])	2,414	21.577
   org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)	2,378	21.255
      org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)	2,378	21.255
         org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table)	2,351	21.014
            org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table, boolean, boolean, Set)	2,351	21.014
               org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient, Set, boolean)	2,350	21.005
                  org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List)	2,326	20.79
                     org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor, HdfsPartition)	2,233	19.959
                        org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition)	1,998	17.858
                           org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path)	1,496	13.371

Issue Links

• duplicates: HADOOP-10768 Optimize Hadoop RPC encryption performance
• is duplicated by: HADOOP-10768 Optimize Hadoop RPC encryption performance