Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14688

Intern strings in KeyVersion and EncryptedKeyVersion

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: kms
    • Labels:
      None
    • Target Version/s:

      Description

      This is inspired by Misha Dmitriev's work on HDFS-11383.

      The key names and key version names are usually the same for a bunch of KeyVersion and EncryptedKeyVersion. We should not create duplicate objects for them.

      This is more important to HDFS-10899, where we try to re-encrypt all files' EDEKs in a given EZ. Those EDEKs all has the same key name, and mostly using no more than a couple of key version names.

      1. GC root of the String.png
        134 kB
        Xiao Chen
      2. HADOOP-14688.01.patch
        2 kB
        Xiao Chen
      3. heapdump analysis.png
        66 kB
        Xiao Chen
      4. jxray.report
        91 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12811/)
          HADOOP-14688. Intern strings in KeyVersion and EncryptedKeyVersion. (weichiu: rev ad32759fd9f33e7bd18758ad1a5a464dab3bcbd9)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderCryptoExtension.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProvider.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12811/ ) HADOOP-14688 . Intern strings in KeyVersion and EncryptedKeyVersion. (weichiu: rev ad32759fd9f33e7bd18758ad1a5a464dab3bcbd9) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderCryptoExtension.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProvider.java
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Wei-Chiu for review and commit. Also thanks Daryn and Misha for commenting.

          Show
          xiaochen Xiao Chen added a comment - Thanks Wei-Chiu for review and commit. Also thanks Daryn and Misha for commenting.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed the patch to branch-2, branch-3.0 and trunk. Thanks Xiao Chen for the patch, and Daryn Sharp Misha Dmitriev for review and thoughts!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed the patch to branch-2, branch-3.0 and trunk. Thanks Xiao Chen for the patch, and Daryn Sharp Misha Dmitriev for review and thoughts!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          +1. will commit today.

          Show
          jojochuang Wei-Chiu Chuang added a comment - +1. will commit today.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks Misha Dmitriev for your insightful comment, and thanks Xiao Chen for the performance test.

          I think the performance optimization makes sense to you especially in the context of key-rotation. Hi Daryn Sharp, how do you think? I would like to cast my +1 if there are no objections.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks Misha Dmitriev for your insightful comment, and thanks Xiao Chen for the performance test. I think the performance optimization makes sense to you especially in the context of key-rotation. Hi Daryn Sharp , how do you think? I would like to cast my +1 if there are no objections.
          Hide
          misha@cloudera.com Misha Dmitriev added a comment -

          Daryn Sharp: when a live heap dump is captured, as done here, a full GC is performed before a heap snapshot is taken. So if the given application produces objects that are very short-lived, i.e. quickly become garbage, then we will only see those of them that are live at the moment, which is typically not much. Conversely, most objects in a live heap dump tend to be relatively long-lived.

          Furthermore, experience has shown that for reasonably long-lived strings, the CPU overhead of interning is small compared to the reduction in the memory pressure, reduced GC pauses, etc. That is, the cost of a fast internal String.intern() call is comparable to the cost of GC scanning and moving around all the extra copies of a string that remain in memory without interning.

          Show
          misha@cloudera.com Misha Dmitriev added a comment - Daryn Sharp : when a live heap dump is captured, as done here, a full GC is performed before a heap snapshot is taken. So if the given application produces objects that are very short-lived, i.e. quickly become garbage, then we will only see those of them that are live at the moment, which is typically not much. Conversely, most objects in a live heap dump tend to be relatively long-lived. Furthermore, experience has shown that for reasonably long-lived strings, the CPU overhead of interning is small compared to the reduction in the memory pressure, reduced GC pauses, etc. That is, the cost of a fast internal String.intern() call is comparable to the cost of GC scanning and moving around all the extra copies of a string that remain in memory without interning.
          Hide
          xiaochen Xiao Chen added a comment - - edited

          Bumping up on this...

          I understand the concern of the added overhead for a normal operation like getFileEncryptionInfo. But from internal runs I did not see this interning causing any visible impact on NN throughput.

          On the other hand, heap is pretty ugly without this one during re-encryption. Attaching a report ran from jxray. The most related section is:

          7. DUPLICATE STRINGS

          Total strings: 2,570,432 Unique strings: 1,033,993 Duplicate values: 3,559 Overhead: 170,572K (8.4%)

          Top duplicate strings:
          Ovhd Num char[]s Num objs Value

          103,775K (5.1%) 830205 830205 "mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf"
          23,042K (1.1%) 184337 184337 "0OFmjElLqXgtjvWKkgfRoLpUj92dHrEaQCPeh3VDh8V"
          8,668K (0.4%) 184937 184937 "EEK"
          2,853K (0.1%) 12176 12176 "POST /kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek?eek_op=reencrypt HTTP/1.1"
          2,473K (0.1%) 12177 12177 "/kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek?eek_op=reencrypt"
          2,298K (0.1%) 13374 13374 "/kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek"

          With the fix in place, the 6.2% goes away.

          Show
          xiaochen Xiao Chen added a comment - - edited Bumping up on this... I understand the concern of the added overhead for a normal operation like getFileEncryptionInfo. But from internal runs I did not see this interning causing any visible impact on NN throughput. On the other hand, heap is pretty ugly without this one during re-encryption. Attaching a report ran from jxray . The most related section is: 7. DUPLICATE STRINGS Total strings: 2,570,432 Unique strings: 1,033,993 Duplicate values: 3,559 Overhead: 170,572K (8.4%) Top duplicate strings: Ovhd Num char[]s Num objs Value 103,775K (5.1%) 830205 830205 "mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf" 23,042K (1.1%) 184337 184337 "0OFmjElLqXgtjvWKkgfRoLpUj92dHrEaQCPeh3VDh8V" 8,668K (0.4%) 184937 184937 "EEK" 2,853K (0.1%) 12176 12176 "POST /kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek?eek_op=reencrypt HTTP/1.1" 2,473K (0.1%) 12177 12177 "/kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek?eek_op=reencrypt" 2,298K (0.1%) 13374 13374 "/kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek" With the fix in place, the 6.2% goes away.
          Hide
          xiaochen Xiao Chen added a comment -

          The heapdumps are too big to attach here, so I uploaded a screenshot of the most relevant analysis result out of it.

          The 2 most duplicated strings (mG... and 0O...) are the 2 key version names. I was running re-encryption on a zone with 1M files. 2 different key versions were among those files in this run.

          Verified after interning, this goes away.

          Daryn Sharp, do you think this makes sense? Thanks!

          Show
          xiaochen Xiao Chen added a comment - The heapdumps are too big to attach here, so I uploaded a screenshot of the most relevant analysis result out of it. The 2 most duplicated strings (mG... and 0O...) are the 2 key version names. I was running re-encryption on a zone with 1M files. 2 different key versions were among those files in this run. Verified after interning, this goes away. Daryn Sharp , do you think this makes sense? Thanks!
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks again Daryn Sharp for the review series!

          True the edeks are stored in a xattr. During re-encryption though, the EDEK object is constructed, and sent to KMS, where a new EDEK is returned.

          What's tricky here is, contacting KMS requires to be done outside of the lock. Therefore, the EDEK object has to exist for that time being. And since we're trying to re-encrypt many EDEKs per batch, there're many on-the-fly EDEK objects. Relative code in ReencryptionHandler$EDEKReencryptCallable#call of HDFS-10899.

          To make things worse, since KMS is proven to be the bottleneck of this, we'd like to multi thread the 'contact KMS' part, which means more on-the-fly EDEKs.... (see multi-threading part of HDFS-10899's doc)

          Show
          xiaochen Xiao Chen added a comment - Thanks again Daryn Sharp for the review series! True the edeks are stored in a xattr. During re-encryption though, the EDEK object is constructed, and sent to KMS, where a new EDEK is returned. What's tricky here is, contacting KMS requires to be done outside of the lock. Therefore, the EDEK object has to exist for that time being. And since we're trying to re-encrypt many EDEKs per batch, there're many on-the-fly EDEK objects. Relative code in ReencryptionHandler$EDEKReencryptCallable#call of HDFS-10899 . To make things worse, since KMS is proven to be the bottleneck of this, we'd like to multi thread the 'contact KMS' part, which means more on-the-fly EDEKs.... (see multi-threading part of HDFS-10899 's doc )
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          +1 mvninstall 13m 45s trunk passed
          +1 compile 14m 18s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 30s trunk passed
          +1 findbugs 1m 33s trunk passed
          +1 javadoc 0m 51s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 42s the patch passed
          +1 compile 11m 8s the patch passed
          +1 javac 11m 8s the patch passed
          +1 checkstyle 0m 38s the patch passed
          +1 mvnsite 1m 30s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 38s the patch passed
          +1 javadoc 0m 52s the patch passed
                Other Tests
          -1 unit 8m 29s hadoop-common in the patch failed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          60m 10s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14688
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879026/HADOOP-14688.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0665569c2ac5 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27a1a5f
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12863/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12863/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12863/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 13m 45s trunk passed +1 compile 14m 18s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 30s trunk passed +1 findbugs 1m 33s trunk passed +1 javadoc 0m 51s trunk passed       Patch Compile Tests +1 mvninstall 0m 42s the patch passed +1 compile 11m 8s the patch passed +1 javac 11m 8s the patch passed +1 checkstyle 0m 38s the patch passed +1 mvnsite 1m 30s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 38s the patch passed +1 javadoc 0m 52s the patch passed       Other Tests -1 unit 8m 29s hadoop-common in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 60m 10s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14688 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879026/HADOOP-14688.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0665569c2ac5 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27a1a5f Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12863/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12863/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12863/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          Where does the interning play a meaningful part in the process of the related jira? Haven't dug into the code, but isn't this information typically encoded in a xattr and only transiently exists during a namesystem operation? If yes, the overhead of interning is likely unworth it. Ie. you already have a unique string, why bother with effectively a hash lookup to sub it with another string if the unique instance is being gc'ed soon anyway.

          Show
          daryn Daryn Sharp added a comment - Where does the interning play a meaningful part in the process of the related jira? Haven't dug into the code, but isn't this information typically encoded in a xattr and only transiently exists during a namesystem operation? If yes, the overhead of interning is likely unworth it. Ie. you already have a unique string, why bother with effectively a hash lookup to sub it with another string if the unique instance is being gc'ed soon anyway.
          Hide
          xiaochen Xiao Chen added a comment -

          Attach a simple patch 1 for this

          Show
          xiaochen Xiao Chen added a comment - Attach a simple patch 1 for this

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development