Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14558

RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: hdfs-client
    • Labels:

      Description

      While running a performance tests for Impala comparing secure and un-secure clusters I noticed that metadata loading operations are 10x slower on a cluster with Kerberos+SSL enabled.
      hadoop.rpc.protection is set to privacy

      Any recommendations on how this can be mitigated? 10x slowdown is a big hit for metadata loading.

      The majority of the slowdown is coming from the two threads below.

      Stack Trace	Sample Count	Percentage(%)
      org.apache.hadoop.ipc.Client$Connection.run()	5,212	46.586
         org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse()	5,203	46.505
            java.io.DataInputStream.readInt()	5,039	45.039
               java.io.BufferedInputStream.read()	5,038	45.03
                  java.io.BufferedInputStream.fill()	5,038	45.03
                     org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int)	5,036	45.013
                        java.io.FilterInputStream.read(byte[], int, int)	5,036	45.013
                           org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int, int)	5,036	45.013
                              org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()	5,035	45.004
                                 com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int)	4,775	42.68
                                    sun.security.jgss.GSSContextImpl.unwrap(byte[], int, int, MessageProp)	4,775	42.68
                                       sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp)	4,768	42.617
                                          sun.security.jgss.krb5.WrapToken.getData()	4,714	42.134
                                             sun.security.jgss.krb5.WrapToken.getData(byte[], int)	4,714	42.134
                                                sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int)	4,714	42.134
                                                   sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int, byte[], int)	3,083	27.556
                                                      sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int, int, byte[], int)	3,078	27.512
                                                         sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[], byte[], int, int)	3,076	27.494
                                                            sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[], byte[], int, int)	3,076	27.494
      

      And

      Stack Trace	Sample Count	Percentage(%)
      java.lang.Thread.run()	3,379	30.202
         java.util.concurrent.ThreadPoolExecutor$Worker.run()	3,379	30.202
            java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)	3,379	30.202
               java.util.concurrent.FutureTask.run()	3,367	30.095
                  java.util.concurrent.Executors$RunnableAdapter.call()	3,367	30.095
                     org.apache.hadoop.ipc.Client$Connection$3.run()	3,367	30.095
                        java.io.DataOutputStream.flush()	3,367	30.095
                           java.io.BufferedOutputStream.flush()	3,367	30.095
                              java.io.BufferedOutputStream.flushBuffer()	3,367	30.095
                                 org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[], int, int)	3,367	30.095
                                    com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int)	3,281	29.326
                                       sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp)	3,281	29.326
                                          sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp)	3,280	29.317
                                             sun.security.jgss.krb5.WrapToken.<init>(Krb5Context, MessageProp, byte[], int, int)	1,872	16.732
                                             sun.security.jgss.krb5.WrapToken.encode()	1,407	12.576
      

      This is the Impala Catalog thread which initiates the NameNode request

      Stack Trace	Sample Count	Percentage(%)
      org.apache.impala.service.JniCatalog.resetMetadata(byte[])	2,414	21.577
         org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)	2,378	21.255
            org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)	2,378	21.255
               org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table)	2,351	21.014
                  org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table, boolean, boolean, Set)	2,351	21.014
                     org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient, Set, boolean)	2,350	21.005
                        org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List)	2,326	20.79
                           org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor, HdfsPartition)	2,233	19.959
                              org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition)	1,998	17.858
                                 org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path)	1,496	13.371
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: