Derby
  1. Derby
  2. DERBY-3710

cannot access a database using AES encryption with encryptionKeyLength=192 after it's been shutdown

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.5.1.1
    • Fix Version/s: 10.5.3.0, 10.6.1.0
    • Component/s: Services
    • Labels:
      None
    • Environment:
      reproduced with ibm's jdk 1.5 and 1.6, and sun's jdk15.
      AES encryption with encryptionKeyLength=192 requires unrestricted security policy jars on your jvm
    • Urgency:
      Normal
    • Issue & fix info:
      Repro attached
    • Bug behavior facts:
      Data corruption

      Description

      Accessing a database created using encryptionAlgorithm: AES/CBC/NoPadding, and encryptionKeyLength=192 after it's been shutdown fails like so:
      -----------------------
      ERROR XJ040: Failed to start database 'encdbcbc_192', see the next exception for details.
      ERROR XBM06: Startup failed. An encrypted database cannot be accessed without the correct boot password.
      ----------------------

      This does not occur when you use encryptionKeyLength=128 (does not require unrestricted jars) nor encryptionKeyLength=256 (does require unrestricted policy jars).

      Note: our test (in derbyall): store/aes.sql does not test this, firstly it doesn't test the larger sizes (because it would diff & fail unless you have been able to adjust your jvm's policy jars), and secondly it doesn't shutdown before reconnecting.

      1. derby-3710-01-aa-digestPaddedPassword.diff
        6 kB
        Rick Hillegas
      2. derby-3710-01-ab-digestPaddedPassword.diff
        6 kB
        Rick Hillegas
      3. repro.sql
        3 kB
        Myrna van Lunteren
      4. repro-3710.sql
        2 kB
        Rick Hillegas

        Issue Links

          Activity

          Hide
          Myrna van Lunteren added a comment -

          attaching an sql script based on store/aes.sql. Run with ij; if you have unrestricted policy jars in place, and the bug occurs, you'll see a failure only with encryptionKeyLength=192.
          (if you don't have the unrestricted policy jars, creation of the database with 192 or 256 length will fail).

          Show
          Myrna van Lunteren added a comment - attaching an sql script based on store/aes.sql. Run with ij; if you have unrestricted policy jars in place, and the bug occurs, you'll see a failure only with encryptionKeyLength=192. (if you don't have the unrestricted policy jars, creation of the database with 192 or 256 length will fail).
          Hide
          Knut Anders Hatlen added a comment -

          Triaged for 10.5.2.

          Show
          Knut Anders Hatlen added a comment - Triaged for 10.5.2.
          Hide
          Rick Hillegas added a comment -

          Attaching a slightly simpler repro-3710.sql. I have verified this behavior on the Apple Java 6 beta for 32-bit Intel machines.

          Show
          Rick Hillegas added a comment - Attaching a slightly simpler repro-3710.sql. I have verified this behavior on the Apple Java 6 beta for 32-bit Intel machines.
          Hide
          Rick Hillegas added a comment -

          I believe that the bug is caused by special logic in JCECipherFactory. For AES encryption, we pad the encryption key to a 16 byte boundary. This means that for AES encryption, a 192 bit key is extended to 256 bits. An MD5 digest is appended to the encrypted key. At initial boot time, this digest is computed on the unpadded value. But on subsequent boots, we check the stored digest against a digest computed on the padded value. The reboot fails because the unpadded digest does not equal the padded digest. I will submit a patch and test for this.

          Show
          Rick Hillegas added a comment - I believe that the bug is caused by special logic in JCECipherFactory. For AES encryption, we pad the encryption key to a 16 byte boundary. This means that for AES encryption, a 192 bit key is extended to 256 bits. An MD5 digest is appended to the encrypted key. At initial boot time, this digest is computed on the unpadded value. But on subsequent boots, we check the stored digest against a digest computed on the padded value. The reboot fails because the unpadded digest does not equal the padded digest. I will submit a patch and test for this.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-3710-01-aa-digestPaddedPassword.diff. This patch fixes encryption to store a digest computed from the padded password rather than the unpadded password. Although this changes what is stored in service.properties, I don't believe that this affects existing applications. That is because this bug would have caused reboot failures in all cases where the value in service.properties has changed. I will run regression tests later today.

          Touches the following files:

          M java/engine/org/apache/derby/impl/services/jce/JCECipherFactory.java

          Store a digest computed from the padded password rather than the unpadded password.

          M java/testing/org/apache/derbyTesting/functionTests/tests/store/EncryptionAESTest.java

          Remove the logic which disabled tests for 192 bit encryption. Those tests had been disabled because of this bug. Now they run again.

          Also add a new test case to make sure that a different version of this bug isn't lurking. The new test case changes the bootpassword, shuts down, and then reboots using the new password.

          Show
          Rick Hillegas added a comment - Attaching derby-3710-01-aa-digestPaddedPassword.diff. This patch fixes encryption to store a digest computed from the padded password rather than the unpadded password. Although this changes what is stored in service.properties, I don't believe that this affects existing applications. That is because this bug would have caused reboot failures in all cases where the value in service.properties has changed. I will run regression tests later today. Touches the following files: M java/engine/org/apache/derby/impl/services/jce/JCECipherFactory.java Store a digest computed from the padded password rather than the unpadded password. M java/testing/org/apache/derbyTesting/functionTests/tests/store/EncryptionAESTest.java Remove the logic which disabled tests for 192 bit encryption. Those tests had been disabled because of this bug. Now they run again. Also add a new test case to make sure that a different version of this bug isn't lurking. The new test case changes the bootpassword, shuts down, and then reboots using the new password.
          Hide
          Dag H. Wanvik added a comment -

          Thanks for this fix, Rick. I had a look at it, and the fix seems reasonable (caveat: unfamiliar code to me). However, I could not make the (old or new) test fail even without the patch to JCECipherFactory - maybe this is just accidental because the digest ignores 0-filled bytes.

          The inner class EncryptedKeyResult should probably be made private.

          Show
          Dag H. Wanvik added a comment - Thanks for this fix, Rick. I had a look at it, and the fix seems reasonable (caveat: unfamiliar code to me). However, I could not make the (old or new) test fail even without the patch to JCECipherFactory - maybe this is just accidental because the digest ignores 0-filled bytes. The inner class EncryptedKeyResult should probably be made private.
          Hide
          Rick Hillegas added a comment -

          Thanks for the quick review, Dag. Attaching a second rev of the patch which makes the tuple inner class private as you suggested: derby-3710-01-ab-digestPaddedPassword.diff.

          I'm not sure why the tests don't work in your environment. One thing I have noticed about this test class is that all sorts of unexpected errors are intercepted in EncryptionAESTest.createAndPopulateDB(). So, for instance, the 192 bit test cases don't run for me under Java 5 because I get an unsupported key length exception on that platform--the test class swallows that exception, skips the test case, and continues merrily along. Have you tried Myrna's repro with and without the patch? Don't bother with my repro: if your platform doesn't support 192 bit encryption, then my repro will fail because of DERBY-4329.

          The full regression tests ran cleanly for me on Java 5 on my Mac-however, as I noted, on that platform the 192 bit test cases are silently ignored. Those test cases do run correctly for me on the beta Java 6 which I'm running on my Mac and, on that platform, the old-style harness tests run cleanly for me too. However, the full JUnit suite hangs for me on that platform-with and without this patch.

          It seems to me that this patch fixes the problem and does not introduce any regressions which I can detect.

          Thanks!

          Show
          Rick Hillegas added a comment - Thanks for the quick review, Dag. Attaching a second rev of the patch which makes the tuple inner class private as you suggested: derby-3710-01-ab-digestPaddedPassword.diff. I'm not sure why the tests don't work in your environment. One thing I have noticed about this test class is that all sorts of unexpected errors are intercepted in EncryptionAESTest.createAndPopulateDB(). So, for instance, the 192 bit test cases don't run for me under Java 5 because I get an unsupported key length exception on that platform--the test class swallows that exception, skips the test case, and continues merrily along. Have you tried Myrna's repro with and without the patch? Don't bother with my repro: if your platform doesn't support 192 bit encryption, then my repro will fail because of DERBY-4329 . The full regression tests ran cleanly for me on Java 5 on my Mac- however, as I noted, on that platform the 192 bit test cases are silently ignored. Those test cases do run correctly for me on the beta Java 6 which I'm running on my Mac and, on that platform, the old-style harness tests run cleanly for me too. However, the full JUnit suite hangs for me on that platform -with and without this patch. It seems to me that this patch fixes the problem and does not introduce any regressions which I can detect. Thanks!
          Hide
          Myrna van Lunteren added a comment -

          for what it's worth...
          The troublesome thing with encryptionAESTest was that encryption lengths > 128 require unrestricted policy files which don't get installed by default, and aren't available to everyone in the world. So there was the choice between introducing false failures for those people who might try to run the tests without them, or risking the tests cases not getting run unnoticed.
          Perhaps we should at least always print out the text indicating the unrestricted policy jar files aren't available (currently it's in a if (TestConfiguration.getCurrent().doTrace()) block, so only will show up if you run with -Dderby.tests.trace=true).

          Note also DERBY-4325 which has a further suggestion for improvement to the encryptionAESTest.

          Show
          Myrna van Lunteren added a comment - for what it's worth... The troublesome thing with encryptionAESTest was that encryption lengths > 128 require unrestricted policy files which don't get installed by default, and aren't available to everyone in the world. So there was the choice between introducing false failures for those people who might try to run the tests without them, or risking the tests cases not getting run unnoticed. Perhaps we should at least always print out the text indicating the unrestricted policy jar files aren't available (currently it's in a if (TestConfiguration.getCurrent().doTrace()) block, so only will show up if you run with -Dderby.tests.trace=true). Note also DERBY-4325 which has a further suggestion for improvement to the encryptionAESTest.
          Hide
          Dag H. Wanvik added a comment - - edited

          Right; thanks Rick and Myrna, that was it apparently:

          with this change in the test:
          :
          @@ -383,6 +388,11 @@
          catch (SQLException e) {
          // if it fails, it should only be because of non-existing
          // support for unrestricted encryption policy.
          + System.err.println("restricted encryption policy: " + algorithm);
          + for (int i=0; i < otherAttributes.length; i++)

          { + System.err.println("attr: " + otherAttributes[i]); + }

          +

          I see:

          java -client -Xms128M -Xmx512M -XX:MaxPermSize=128m -DderbyTesting.oldReleasePath=/usr/local/share/java/derby/lib-debug junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.store.EncryptionAESTest
          ...restricted encryption policy: AES/CBC/NoPadding
          attr: encryptionKeyLength=192
          attr: bootPassword=Thursday
          restricted encryption policy: AES/CBC/NoPadding
          attr: encryptionKeyLength=256
          attr: bootPassword=Thursday

          so I guess what I saw is expected. I guess I can't get hold of the unrestricted policy jars?

          Show
          Dag H. Wanvik added a comment - - edited Right; thanks Rick and Myrna, that was it apparently: with this change in the test: : @@ -383,6 +388,11 @@ catch (SQLException e) { // if it fails, it should only be because of non-existing // support for unrestricted encryption policy. + System.err.println("restricted encryption policy: " + algorithm); + for (int i=0; i < otherAttributes.length; i++) { + System.err.println("attr: " + otherAttributes[i]); + } + I see: java -client -Xms128M -Xmx512M -XX:MaxPermSize=128m -DderbyTesting.oldReleasePath=/usr/local/share/java/derby/lib-debug junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.store.EncryptionAESTest ...restricted encryption policy: AES/CBC/NoPadding attr: encryptionKeyLength=192 attr: bootPassword=Thursday restricted encryption policy: AES/CBC/NoPadding attr: encryptionKeyLength=256 attr: bootPassword=Thursday so I guess what I saw is expected. I guess I can't get hold of the unrestricted policy jars?
          Hide
          Rick Hillegas added a comment -

          Hi Dag,

          Have you tried downloading and installing the "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 6" from http://java.sun.com/javase/downloads/index.jsp ? Are you not allowed to use this strong encryption because you live in Norway?

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Hi Dag, Have you tried downloading and installing the "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 6" from http://java.sun.com/javase/downloads/index.jsp ? Are you not allowed to use this strong encryption because you live in Norway? Thanks, -Rick
          Hide
          Rick Hillegas added a comment -

          Hi Myrna,

          Could you verify whether this patch fixes the problem? Thanks!

          Show
          Rick Hillegas added a comment - Hi Myrna, Could you verify whether this patch fixes the problem? Thanks!
          Hide
          Dag H. Wanvik added a comment -

          Rick, I was able to download it now. I just never realized I needed to download separately to be able to run all
          tests. Should it mentioned in testing/README.htm?

          Show
          Dag H. Wanvik added a comment - Rick, I was able to download it now. I just never realized I needed to download separately to be able to run all tests. Should it mentioned in testing/README.htm?
          Hide
          Rick Hillegas added a comment -

          Thanks, Dag, I think that is a good suggestion.

          Show
          Rick Hillegas added a comment - Thanks, Dag, I think that is a good suggestion.
          Hide
          Myrna van Lunteren added a comment -

          Hi,

          I confirmed that with the patch installed, the problem hit by the repros is no longer present. The patch looks good to me. (I tested with IBM1.6 SR5 with unrestricted policy jars installed).

          Show
          Myrna van Lunteren added a comment - Hi, I confirmed that with the patch installed, the problem hit by the repros is no longer present. The patch looks good to me. (I tested with IBM1.6 SR5 with unrestricted policy jars installed).
          Hide
          Rick Hillegas added a comment -

          Thanks Dag and Myrna. I have committed derby-3710-01-ab-digestPaddedPassword.diff at subversion revision 800773.

          Show
          Rick Hillegas added a comment - Thanks Dag and Myrna. I have committed derby-3710-01-ab-digestPaddedPassword.diff at subversion revision 800773.
          Hide
          Rick Hillegas added a comment -

          Ported revision 800773 from trunk to 10.5 branch at revision 800774.

          Show
          Rick Hillegas added a comment - Ported revision 800773 from trunk to 10.5 branch at revision 800774.
          Hide
          Dag H. Wanvik added a comment -

          I have added a new section to platform specific hints on how to run JUnit tests:

          http://wiki.apache.org/db-derby/JunitVmIssues

          called "Sun Java 1.4.2, 1.5 and 1.6: unlimited cryptography policies" so testers are able to install what's needed to run the complete set of encryption tests.

          Show
          Dag H. Wanvik added a comment - I have added a new section to platform specific hints on how to run JUnit tests: http://wiki.apache.org/db-derby/JunitVmIssues called "Sun Java 1.4.2, 1.5 and 1.6: unlimited cryptography policies" so testers are able to install what's needed to run the complete set of encryption tests.

            People

            • Assignee:
              Rick Hillegas
              Reporter:
              Myrna van Lunteren
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development