Flume
  1. Flume
  2. FLUME-1789

Unit tests TestJCEFileKeyProvider and TestFileChannelEncryption fail with IBM JDK and flume-1.3.0

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.3.0
    • Fix Version/s: v1.4.0, v1.3.1
    • Component/s: Channel, Test
    • Environment:

      IBM Java 6 SR 12
      RHEL 6.3

      Description

      The unit tests TestJCEFileKeyProvider and TestFileChannelEncryption fail
      when running with IBM Java 6. Only the test cases that use an existing
      keystore file fail. It seems like the problem is that IBM keytool can't read
      the file test.keystore provided with flume (which was probably generated
      with SUN's keytool). Replacing the file test.keystore (at
      flume-ng-channels/flume-file-channel/src/test/resources) with one generated
      by IBM keytool fixes the issue (and it also works with SUN Java 6).

      1. test.keystore
        0.7 kB
        Aline Guedes Pinto
      2. sun-testkeystore
        0.9 kB
        Aline Guedes Pinto
      3. ibm-testkeystore
        0.7 kB
        Aline Guedes Pinto
      4. FLUME-1789.patch
        2 kB
        Aline Guedes Pinto

        Activity

        Hide
        Hudson added a comment -

        Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/)
        FLUME-1789: Unit tests TestJCEFileKeyProvider and TestFileChannelEncryption fail with IBM JDK and flume-1.3.0 (Revision 159f0bd26e80ca76bd6ff80901f9e6cc14efbc66)

        Result = SUCCESS
        brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=159f0bd26e80ca76bd6ff80901f9e6cc14efbc66
        Files :

        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/EncryptionTestUtils.java
        • flume-ng-channels/flume-file-channel/src/test/resources/sun-test.keystore
        • flume-ng-channels/flume-file-channel/src/test/resources/test.keystore
        • flume-ng-channels/flume-file-channel/src/test/resources/ibm-test.keystore
        Show
        Hudson added a comment - Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/ ) FLUME-1789 : Unit tests TestJCEFileKeyProvider and TestFileChannelEncryption fail with IBM JDK and flume-1.3.0 (Revision 159f0bd26e80ca76bd6ff80901f9e6cc14efbc66) Result = SUCCESS brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=159f0bd26e80ca76bd6ff80901f9e6cc14efbc66 Files : flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/EncryptionTestUtils.java flume-ng-channels/flume-file-channel/src/test/resources/sun-test.keystore flume-ng-channels/flume-file-channel/src/test/resources/test.keystore flume-ng-channels/flume-file-channel/src/test/resources/ibm-test.keystore
        Hide
        Brock Noland added a comment -

        Aline, I committed this to trunk and 1.4! Thank you for your contribution!

        Show
        Brock Noland added a comment - Aline, I committed this to trunk and 1.4! Thank you for your contribution!
        Hide
        Aline Guedes Pinto added a comment -

        Attached patch to fix the issue as per Brock' suggestion.

        Replaced test.keystore with sun-testkeystore and created a new ibm-testkeystore file.

        Tested with IBM and SUN java:
        [alinegds@oc1023547534 flume]$ java -version
        java version "1.6.0"
        Java(TM) SE Runtime Environment (build pxa6460sr11-20120806_01(SR11))
        IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr11-20120801_118201 (JIT enabled, AOT enabled)
        J9VM - 20120801_118201
        JIT - r9_20120608_24176ifx1
        GC - 20120516_AA)
        JCL - 20120713_01

        [alinegds@oc1023547534 flume]$ java -version
        java version "1.6.0_38"
        Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
        Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)

        Show
        Aline Guedes Pinto added a comment - Attached patch to fix the issue as per Brock' suggestion. Replaced test.keystore with sun-testkeystore and created a new ibm-testkeystore file. Tested with IBM and SUN java: [alinegds@oc1023547534 flume] $ java -version java version "1.6.0" Java(TM) SE Runtime Environment (build pxa6460sr11-20120806_01(SR11)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr11-20120801_118201 (JIT enabled, AOT enabled) J9VM - 20120801_118201 JIT - r9_20120608_24176ifx1 GC - 20120516_AA) JCL - 20120713_01 [alinegds@oc1023547534 flume] $ java -version java version "1.6.0_38" Java(TM) SE Runtime Environment (build 1.6.0_38-b05) Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)
        Hide
        Brock Noland added a comment -

        I get the error below on my mac with:

        $ java -version
        java version "1.6.0_37"
        Java(TM) SE Runtime Environment (build 1.6.0_37-b06-434-11M3909)
        Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01-434, mixed mode)

        How about we just select the keystore based on the JVM? Ie sun-test.keystore and ibm-test.keystore.

        estWithExistingKeyStore(org.apache.flume.channel.file.encryption.TestJCEFileKeyProvider)  Time elapsed: 0.415 sec  <<< ERROR!
        java.lang.RuntimeException: java.security.UnrecoverableKeyException: com.ibm.crypto.provider.AESSecretKey. Key = key-0, passwordFile = /var/folders/6l/2kf3r2pj1t176h2nhdwfpk1r0000gp/T/1355854596783-0/key-0
                at org.apache.flume.channel.file.encryption.JCEFileKeyProvider.getKey(JCEFileKeyProvider.java:82)
                at org.apache.flume.channel.file.encryption.TestJCEFileKeyProvider.testKeyProvider(TestJCEFileKeyProvider.java:107)
                at org.apache.flume.channel.file.encryption.TestJCEFileKeyProvider.testWithExistingKeyStore(TestJCEFileKeyProvider.java:92)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
                at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
                at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
                at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
                at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
                at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
                at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
                at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
                at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
                at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
                at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
                at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
                at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
                at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:228)
                at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
                at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
                at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
                at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
                at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
                at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
        Caused by: java.security.UnrecoverableKeyException: com.ibm.crypto.provider.AESSecretKey
                at com.sun.crypto.provider.SunJCE_z.a(DashoA13*..)
                at com.sun.crypto.provider.JceKeyStore.engineGetKey(DashoA13*..)
                at java.security.KeyStore.getKey(KeyStore.java:763)
        
        Show
        Brock Noland added a comment - I get the error below on my mac with: $ java -version java version "1.6.0_37" Java(TM) SE Runtime Environment (build 1.6.0_37-b06-434-11M3909) Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01-434, mixed mode) How about we just select the keystore based on the JVM? Ie sun-test.keystore and ibm-test.keystore. estWithExistingKeyStore(org.apache.flume.channel.file.encryption.TestJCEFileKeyProvider) Time elapsed: 0.415 sec <<< ERROR! java.lang.RuntimeException: java.security.UnrecoverableKeyException: com.ibm.crypto.provider.AESSecretKey. Key = key-0, passwordFile = /var/folders/6l/2kf3r2pj1t176h2nhdwfpk1r0000gp/T/1355854596783-0/key-0 at org.apache.flume.channel.file.encryption.JCEFileKeyProvider.getKey(JCEFileKeyProvider.java:82) at org.apache.flume.channel.file.encryption.TestJCEFileKeyProvider.testKeyProvider(TestJCEFileKeyProvider.java:107) at org.apache.flume.channel.file.encryption.TestJCEFileKeyProvider.testWithExistingKeyStore(TestJCEFileKeyProvider.java:92) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:228) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) Caused by: java.security.UnrecoverableKeyException: com.ibm.crypto.provider.AESSecretKey at com.sun.crypto.provider.SunJCE_z.a(DashoA13*..) at com.sun.crypto.provider.JceKeyStore.engineGetKey(DashoA13*..) at java.security.KeyStore.getKey(KeyStore.java:763)
        Hide
        Aline Guedes Pinto added a comment -

        Attached test.keystore file generated with IBM keytool with the commands below:

        keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES -keysize 128 -validity 9000 -keystore test.keystore -storetype jceks -storepass keyStorePassword
        keytool -genseckey -alias key-1 -keyalg AES -keysize 128 -validity 9000 -keystore test.keystore -storetype jceks -storepass keyStorePassword

        Show
        Aline Guedes Pinto added a comment - Attached test.keystore file generated with IBM keytool with the commands below: keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES -keysize 128 -validity 9000 -keystore test.keystore -storetype jceks -storepass keyStorePassword keytool -genseckey -alias key-1 -keyalg AES -keysize 128 -validity 9000 -keystore test.keystore -storetype jceks -storepass keyStorePassword

          People

          • Assignee:
            Aline Guedes Pinto
            Reporter:
            Aline Guedes Pinto
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development