Derby
  1. Derby
  2. DERBY-5676

Cannot connect to AES encrypted database with IBMJCECCA provider

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Blocker Blocker
    • Resolution: Unresolved
    • Affects Version/s: 10.5.3.0
    • Fix Version/s: None
    • Component/s: Services
    • Environment:
    • Urgency:
      Normal

      Description

      The install is created on a HFS file system. And we create DB using this command
      java -cp . -jar derbyrun.jar ij databaseAuth.sql
      and databaseAuth.sql content is given below.

      connect 'jdbc:derby: /HO4/test/testDb;create=true;dataEncryption=true;bootPassword=Password;encryptionAlgorithm=AES/CBC/NoPadding;';

      --------------------------------------------------------------------------------
      – CREATE TESTTABLE TABLE WITH PRIMARY KEY OF USERNAME
      – STEP 2
      --------------------------------------------------------------------------------
      CREATE TABLE TESTTABLE
      (
      USERNAME VARCHAR(30) NOT NULL,
      PASSWORD VARCHAR(30) NOT NULL,
      PRIMARY KEY (USERNAME)
      );

      --------------------------------------------------------------------------------
      – INSERT USER INTO TESTTABLE TABLE
      – STEP 3
      --------------------------------------------------------------------------------
      INSERT INTO TESTTABLE VALUES('<adminuser>', '<adminpass>');

      EXIT;

      Create was successful. But when we try to connect to DB again it fails to start with error.

      1. scriptinascii.zip
        7 kB
        Ambili
      2. db.tar
        3.49 MB
        Ambili
      3. scripts.tar
        50 kB
        Ambili
      4. db.tar
        1.75 MB
        Kathey Marsden
      5. db.pax.Z
        1.78 MB
        Ambili
      6. derby.log
        22 kB
        Ambili

        Activity

        Hide
        Ambili added a comment -

        Derby log attached. The File system on which the derby is created is HFS

        Show
        Ambili added a comment - Derby log attached. The File system on which the derby is created is HFS
        Hide
        Kathey Marsden added a comment -

        Hello Ambili,

        I tried your script on a z/os machine I have access to but not HFS, (changing the database name to a local db and removing the extraneous single quote) and was able to reconnect and recover.

        Is it possible HFS has some extreme write caching? Can you post the resulting corrupt database?

        Show
        Kathey Marsden added a comment - Hello Ambili, I tried your script on a z/os machine I have access to but not HFS, (changing the database name to a local db and removing the extraneous single quote) and was able to reconnect and recover. Is it possible HFS has some extreme write caching? Can you post the resulting corrupt database?
        Hide
        Ambili added a comment -

        The Pax file attached has the database that was getting corrupted. The derby.log is also in the pax. if you have access to Z/os. Download the file and do a
        pax -rvf db.pax.Z to extract the file

        Show
        Ambili added a comment - The Pax file attached has the database that was getting corrupted. The derby.log is also in the pax. if you have access to Z/os. Download the file and do a pax -rvf db.pax.Z to extract the file
        Hide
        Ambili added a comment -

        Hi Kathey,

        Thanks for trying it out. I have the script working on other machines where java version is different. I am not sure whether the issue is with File System settings. If you know any related information, please let me know. Appreciate your help.

        thanks
        Ambili

        Show
        Ambili added a comment - Hi Kathey, Thanks for trying it out. I have the script working on other machines where java version is different. I am not sure whether the issue is with File System settings. If you know any related information, please let me know. Appreciate your help. thanks Ambili
        Hide
        Mike Matrigali added a comment -

        can you show exact way you are trying to connect again?

        Do you have same problem if you don't encrypt? am wondering if maybe you are mismatching encryption somehow.

        Show
        Mike Matrigali added a comment - can you show exact way you are trying to connect again? Do you have same problem if you don't encrypt? am wondering if maybe you are mismatching encryption somehow.
        Hide
        Mike Matrigali added a comment -

        maybe post your exact script to create db and to reconnect. Maybe some wierdness in the quoting, your post to the eye looks like it has wrong single quotes.

        Show
        Mike Matrigali added a comment - maybe post your exact script to create db and to reconnect. Maybe some wierdness in the quoting, your post to the eye looks like it has wrong single quotes.
        Hide
        Kathey Marsden added a comment -

        Also please post the exact urls for the connection on create and then the attempted reconnect. I am wondering if there is some mismatch of encryption because of the few odd characters in the url.

        Show
        Kathey Marsden added a comment - Also please post the exact urls for the connection on create and then the attempted reconnect. I am wondering if there is some mismatch of encryption because of the few odd characters in the url.
        Hide
        Ambili added a comment - - edited

        First i was trying to connect from tomcat. But later we tried creating a sample database ( attached here) and tried to connect using ij tool.

        This is what is used to test the connection using ij tool. the Sql file to used in ij tool is

        connect 'jdbc:derby:/HO43/test/testDb;dataEncryption=true;bootPassword=<password>;encryptionAlgorithm=AES/CBC/NoPadding;';

        SELECT * FROM TESTTABLE;

        And sorry about the typo in the old post. The create query is

        connect 'jdbc:derby:/HO43/test/testDb;create=true;dataEncryption=true;bootPassword=<password>;encryptionAlgorithm=AES/CBC/NoPadding;';

        Show
        Ambili added a comment - - edited First i was trying to connect from tomcat. But later we tried creating a sample database ( attached here) and tried to connect using ij tool. This is what is used to test the connection using ij tool. the Sql file to used in ij tool is connect 'jdbc:derby:/HO43/test/testDb;dataEncryption=true;bootPassword=<password>;encryptionAlgorithm=AES/CBC/NoPadding;'; SELECT * FROM TESTTABLE; And sorry about the typo in the old post. The create query is connect 'jdbc:derby:/HO43/test/testDb;create=true;dataEncryption=true;bootPassword=<password>;encryptionAlgorithm=AES/CBC/NoPadding;';
        Hide
        Kathey Marsden added a comment -

        Can you reproduce using ij to both create and reconnect?

        Show
        Kathey Marsden added a comment - Can you reproduce using ij to both create and reconnect?
        Hide
        Kathey Marsden added a comment -

        Here is a tarred database which might be more convenient on other platforms. I see the NullPointerException on both z/os and windows connecting to this database.

        Show
        Kathey Marsden added a comment - Here is a tarred database which might be more convenient on other platforms. I see the NullPointerException on both z/os and windows connecting to this database.
        Hide
        Ambili added a comment -

        kathey,

        yes. We can reproduce that with ij tool. The script I posted are using ij. The first one to create Db, create a table and one record in that table.

        The second script to query the database created , and select the row in the table USER_CREDENTIALS.
        ambili

        Show
        Ambili added a comment - kathey, yes. We can reproduce that with ij tool. The script I posted are using ij. The first one to create Db, create a table and one record in that table. The second script to query the database created , and select the row in the table USER_CREDENTIALS. ambili
        Hide
        Kathey Marsden added a comment -

        Thanks Ambili,

        I am unable to reproduce here, so the issue must need your disk or configuration.
        Could you please
        1) attach the exact create and select scripts you use to reproduce.
        2) Check with your system programer and see if there is any sort of caching that occurs with your disk configuration.
        3) Try with a 10.8.2.2 debug build.

        Show
        Kathey Marsden added a comment - Thanks Ambili, I am unable to reproduce here, so the issue must need your disk or configuration. Could you please 1) attach the exact create and select scripts you use to reproduce. 2) Check with your system programer and see if there is any sort of caching that occurs with your disk configuration. 3) Try with a 10.8.2.2 debug build.
        Hide
        Mike Matrigali added a comment -

        do you get same problems if you don't use encryption?

        do you get same problem with other types of encryption?

        Can you verify that /HO43/Vantagegmi/webclientdb/VantageDb directory does not exist before running the create script?

        Show
        Mike Matrigali added a comment - do you get same problems if you don't use encryption? do you get same problem with other types of encryption? Can you verify that /HO43/Vantagegmi/webclientdb/VantageDb directory does not exist before running the create script?
        Hide
        Ambili added a comment -

        Hi Kathey,
        I am attaching the scripts I used to create and query the db.

        The output of RunScript.sh is in script.out. The CreateVantageDB.sh uses the databaseAuth_template.sql to create the DB and the DBQuery.sh query the database.

        All the script files are in ebcdic format.

        I will check with system programmer to see what sort of caching is enabled. Do you know how can this be checked against filesystem?

        thanks
        Ambili

        Show
        Ambili added a comment - Hi Kathey, I am attaching the scripts I used to create and query the db. The output of RunScript.sh is in script.out. The CreateVantageDB.sh uses the databaseAuth_template.sql to create the DB and the DBQuery.sh query the database. All the script files are in ebcdic format. I will check with system programmer to see what sort of caching is enabled. Do you know how can this be checked against filesystem? thanks Ambili
        Hide
        Ambili added a comment -

        The db.tar contains VantageDb and VantageDb1. This is the output of the RunScript.sh from the scripts.tar

        VantageDb1 is the backed up DB after the creation
        VantageDb is the DB after DBQuery.sh is ran

        Show
        Ambili added a comment - The db.tar contains VantageDb and VantageDb1. This is the output of the RunScript.sh from the scripts.tar VantageDb1 is the backed up DB after the creation VantageDb is the DB after DBQuery.sh is ran
        Hide
        Mike Matrigali added a comment -

        i don't know how to read ebcdic format on my windows machine. Since we are having a hard time reproducing I wonder if ebcidic is key part of the problem, maybe something is getting confused between ij and the server.

        Show
        Mike Matrigali added a comment - i don't know how to read ebcdic format on my windows machine. Since we are having a hard time reproducing I wonder if ebcidic is key part of the problem, maybe something is getting confused between ij and the server.
        Hide
        Ambili added a comment -

        I am attaching the same scripts and output of the scripts in Ascii format for reading on windows

        Show
        Ambili added a comment - I am attaching the same scripts and output of the scripts in Ascii format for reading on windows
        Hide
        Ambili added a comment -

        Mike,

        The issue is not with ebcidic format since the same script work on other system.
        The script ran is RunScript.sh which creates a DB at the user given path and does a backup of the db and then query the DB. You can check the script.out file in the attached zip to get the output of ij command.

        regards
        Ambili

        Show
        Ambili added a comment - Mike, The issue is not with ebcidic format since the same script work on other system. The script ran is RunScript.sh which creates a DB at the user given path and does a backup of the db and then query the DB. You can check the script.out file in the attached zip to get the output of ij command. regards Ambili
        Hide
        Myrna van Lunteren added a comment -

        I thought I had seen before that you said this worked on another system, but couldn't find that comment again.
        How many z/OS systems do you have access to? Is this one the only one where you see this failure?
        Kathey has tried your script on a z/OS system (with a slightly different jvm version) and it worked fine there too.

        It seems indicated that there is a difference between this system and the other one, or things would work the same way. We need to find that difference to move forward.

        Things I can think of that might be relevant:

        • permissions/rights for the user running the script
        • file permissions (read/write) after creation of the database between the two systems
        • type file system
        • jvm version
        • OS version
        • path settings for the user running the script
        • other derby installations present?
          Double checking these settings are the same on the two systems would be helpful.

        HTH
        Myrna

        Show
        Myrna van Lunteren added a comment - I thought I had seen before that you said this worked on another system, but couldn't find that comment again. How many z/OS systems do you have access to? Is this one the only one where you see this failure? Kathey has tried your script on a z/OS system (with a slightly different jvm version) and it worked fine there too. It seems indicated that there is a difference between this system and the other one, or things would work the same way. We need to find that difference to move forward. Things I can think of that might be relevant: permissions/rights for the user running the script file permissions (read/write) after creation of the database between the two systems type file system jvm version OS version path settings for the user running the script other derby installations present? Double checking these settings are the same on the two systems would be helpful. HTH Myrna
        Hide
        Kathey Marsden added a comment -

        Ambili,
        I have just been working on another issue, DERBY-5685 which seems to be specific to z196 and won't reproduce on my z10 system. Do you have z196 or z10?

        Show
        Kathey Marsden added a comment - Ambili, I have just been working on another issue, DERBY-5685 which seems to be specific to z196 and won't reproduce on my z10 system. Do you have z196 or z10?
        Hide
        Ambili added a comment -

        Hi Kathey,

        This is a customer issue. I cannot check it here. I will check with customer and let you know. I see that the in logs java system properties print os version like this.
        os.version=01.13.00

        Show
        Ambili added a comment - Hi Kathey, This is a customer issue. I cannot check it here. I will check with customer and let you know. I see that the in logs java system properties print os version like this. os.version=01.13.00
        Hide
        Kathey Marsden added a comment -

        I got some information on checking the hardware and also disabling z196 specific optimization if needed.
        uname -Iarns shows a four digit number:
        2817 is z196
        2097 is z10
        2094 is z9

        -Xjit:disableZGryphon disables z196 specific JIT optimizations.
        I don't actually think this is jit as it is not iterative but just putting that info here for completeness.

        Show
        Kathey Marsden added a comment - I got some information on checking the hardware and also disabling z196 specific optimization if needed. uname -Iarns shows a four digit number: 2817 is z196 2097 is z10 2094 is z9 -Xjit:disableZGryphon disables z196 specific JIT optimizations. I don't actually think this is jit as it is not iterative but just putting that info here for completeness.
        Hide
        Stan Bradbury added a comment -

        I don't see where the issue of whether this file system mount utilizes network software or not. I am focusing on the corruption aspect alone, not encryption plays a part. I found the followin reference at:
        http://publib.boulder.ibm.com/infocenter/zos/basics/index.jsp?topic=/com.ibm.zos.zconcepts/zconcepts_177.htm
        TITLE: z/OS concepts:
        Some of these are network mounts

        You can use the following file system types with z/OS UNIX:

        zSeries® File System (zFS), which is a file system that stores files in VSAM linear data sets.
        Hierarchical file system (HFS), a mountable file system, which is being phased out by zFS.
        z/OS Network File System (z/OS NFS), which allows a z/OS system to access a remote UNIX (z/OS or non-z/OS) file system over TCP/IP, as if it were part of the local z/OS directory tree.
        Temporary file system (TFS), which is a temporary, in-memory physical file system that supports in-storage mountable file systems.

        Show
        Stan Bradbury added a comment - I don't see where the issue of whether this file system mount utilizes network software or not. I am focusing on the corruption aspect alone, not encryption plays a part. I found the followin reference at: http://publib.boulder.ibm.com/infocenter/zos/basics/index.jsp?topic=/com.ibm.zos.zconcepts/zconcepts_177.htm TITLE: z/OS concepts: Some of these are network mounts You can use the following file system types with z/OS UNIX: zSeries® File System (zFS), which is a file system that stores files in VSAM linear data sets. Hierarchical file system (HFS), a mountable file system, which is being phased out by zFS. z/OS Network File System (z/OS NFS), which allows a z/OS system to access a remote UNIX (z/OS or non-z/OS) file system over TCP/IP, as if it were part of the local z/OS directory tree. Temporary file system (TFS), which is a temporary, in-memory physical file system that supports in-storage mountable file systems.
        Hide
        Ambili added a comment -

        is there any update on this issue? We were able to get this resolved by not using encryption.
        Is there any workaround I can do to still use encruption but not have the error we were getting?

        Ambili

        Show
        Ambili added a comment - is there any update on this issue? We were able to get this resolved by not using encryption. Is there any workaround I can do to still use encruption but not have the error we were getting? Ambili
        Hide
        Bryan Pendleton added a comment -

        At the very least, it seems like we should document the incompatibility between IBMJCECCA
        encryption provider and Derby's encryption requests.

        Ideally, if we could understand the incompatibility more clearly, it would be nice to
        modify Derby's encryption code to be compatible with the IBMJCECCA provider.

        Show
        Bryan Pendleton added a comment - At the very least, it seems like we should document the incompatibility between IBMJCECCA encryption provider and Derby's encryption requests. Ideally, if we could understand the incompatibility more clearly, it would be nice to modify Derby's encryption code to be compatible with the IBMJCECCA provider.
        Hide
        Knut Anders Hatlen added a comment -

        Adjusted fields for the 10.10 bug triage:

        • Clarified summary
        • Reset urgency to Normal
        • Changed component from Tools to Services
        Show
        Knut Anders Hatlen added a comment - Adjusted fields for the 10.10 bug triage: Clarified summary Reset urgency to Normal Changed component from Tools to Services

          People

          • Assignee:
            Unassigned
            Reporter:
            Ambili
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development