Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.1.2.1, 10.2.2.0, 10.3.1.4, 10.4.1.3
    • Fix Version/s: 10.3.1.4, 10.4.1.3, 10.6.2.1, 10.7.1.1
    • Component/s: Tools
    • Labels:
      None
    • Issue & fix info:
      Release Note Needed
    • Bug behavior facts:
      Security

      Description

      Export should not overwrite existing files, but rather insist that the user remove them before writing to the file. This will help prevent accidental or intentional corruption of the database with export. This may introduce a compatibility issue with export but because export is usually an attended utility and not typically invoked as part of an application, I think the risk is worth the additional security this will provide.

      1. releaseNotev0.html
        5 kB
        Ramin Moazeni
      2. releaseNote.html
        5 kB
        Rick Hillegas
      3. DERBY-2925v6.stat
        0.2 kB
        Ramin Moazeni
      4. DERBY-2925v6.diff
        3 kB
        Ramin Moazeni
      5. DERBY-2925v5.stat
        0.1 kB
        Ramin Moazeni
      6. DERBY-2925v5.diff
        1 kB
        Ramin Moazeni
      7. DERBY-2925v4.stat
        0.2 kB
        Ramin Moazeni
      8. DERBY-2925v4.diff
        3 kB
        Ramin Moazeni
      9. DERBY-2925v3.stat
        1 kB
        Ramin Moazeni
      10. DERBY-2925v3.diff
        68 kB
        Ramin Moazeni
      11. DERBY-2925v2.stat
        1 kB
        Ramin Moazeni
      12. DERBY-2925v2.diff
        57 kB
        Ramin Moazeni
      13. DERBY-2925v1.stat
        1.0 kB
        Ramin Moazeni
      14. DERBY-2925v1.diff
        45 kB
        Ramin Moazeni
      15. DERBY-2925v0.stat
        0.3 kB
        Ramin Moazeni
      16. DERBY-2925v0.diff
        6 kB
        Ramin Moazeni
      17. derby-2925-07-aa-fileUrl.diff
        5 kB
        Rick Hillegas

        Issue Links

          Activity

          Hide
          Ramin Moazeni added a comment -

          To Reproduce this issue:
          ij> connect 'jdbc:derby:test1;create=true';
          ij> create table ex_emp(id int , name char(7) , skills varchar(200), salary decimal(10,2));
          ij> insert into ex_emp values(99,'smith','tennis"p,l,ayer"',190.55);
          ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE (null, 'EX_EMP' , '/home/ramin/emp.dat', null, null, null);
          [ramin@ramin ~]$ ls -ltr emp.dat
          rw-rr- 1 ramin ramin 43 Jul 12 04:57 emp.dat

          Calling SYSCS_UTIL.SYSCS_EXPORT_TABLE for a second time:
          ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE (null, 'EX_EMP' , '/home/ramin/emp.dat', null, null, null);
          [ramin@ramin ~]$ ls -ltr emp.dat
          rw-rr- 1 ramin ramin 43 Jul 12 05:04 emp.dat

          As you can see, the problem is reproduced this through the ij tool. I have yet to write a program
          for this...but I think it is mentioned that this won't be invoked from an application.

          Show
          Ramin Moazeni added a comment - To Reproduce this issue: ij> connect 'jdbc:derby:test1;create=true'; ij> create table ex_emp(id int , name char(7) , skills varchar(200), salary decimal(10,2)); ij> insert into ex_emp values(99,'smith','tennis"p,l,ayer"',190.55); ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE (null, 'EX_EMP' , '/home/ramin/emp.dat', null, null, null); [ramin@ramin ~] $ ls -ltr emp.dat rw-r r - 1 ramin ramin 43 Jul 12 04:57 emp.dat Calling SYSCS_UTIL.SYSCS_EXPORT_TABLE for a second time: ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE (null, 'EX_EMP' , '/home/ramin/emp.dat', null, null, null); [ramin@ramin ~] $ ls -ltr emp.dat rw-r r - 1 ramin ramin 43 Jul 12 05:04 emp.dat As you can see, the problem is reproduced this through the ij tool. I have yet to write a program for this...but I think it is mentioned that this won't be invoked from an application.
          Hide
          Kathey Marsden added a comment -

          I think it is fine to reproduce with IJ and update importExportThruIJ.sql to test your change.
          The tests e.g. ImportExportBaseTest actually do call the procedure from a java program. It would be good to add a test there too.

          Show
          Kathey Marsden added a comment - I think it is fine to reproduce with IJ and update importExportThruIJ.sql to test your change. The tests e.g. ImportExportBaseTest actually do call the procedure from a java program. It would be good to add a test there too.
          Hide
          Ramin Moazeni added a comment - - edited

          Hello,

          I am attaching an interim patch for this issue. I am still working on adding
          the tests to ImportExportBaseTest.
          Your review and feedback is greatly appreciated.

          To test the patch:
          1) ij> connect 'jdbc:derby:test1;create=true';
          2) create table ex_emp(id int , name char(7) , skills varchar(200), salary decimal(10,2));
          3) ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE (null, 'EX_EMP' , '/home/ramin/emp.dat', null, null, null);
          4) ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE (null, 'EX_EMP' , '/home/ramin/emp.dat', null, null, null);
          ERROR XSDF0: Could not create file /home/ramin/emp.dat as it already exists.
          5) ij> CALL SYSCS_UTIL.SYSCS_EXPORT_QUERY_LOBS_TO_EXTFILE('SELECT * FROM EX_EMP','emp.del', ',' ,'"','UTF-8','pictures.dat');
          0 rows inserted/updated/deleted
          6) ij> CALL SYSCS_UTIL.SYSCS_EXPORT_QUERY_LOBS_TO_EXTFILE('SELECT * FROM EX_EMP','emp.del', ',' ,'"','UTF-8','pictures.dat');
          ERROR XSDF0: Could not create file emp.del as it already exists.
          7) ij> CALL SYSCS_UTIL.SYSCS_EXPORT_QUERY_LOBS_TO_EXTFILE('SELECT * FROM EX_EMP','staff.del', ',' ,'"','UTF-8','pictures.dat');
          ERROR XSDF0: Could not create file pictures.dat as it already exists.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - - edited Hello, I am attaching an interim patch for this issue. I am still working on adding the tests to ImportExportBaseTest. Your review and feedback is greatly appreciated. To test the patch: 1) ij> connect 'jdbc:derby:test1;create=true'; 2) create table ex_emp(id int , name char(7) , skills varchar(200), salary decimal(10,2)); 3) ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE (null, 'EX_EMP' , '/home/ramin/emp.dat', null, null, null); 4) ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE (null, 'EX_EMP' , '/home/ramin/emp.dat', null, null, null); ERROR XSDF0: Could not create file /home/ramin/emp.dat as it already exists. 5) ij> CALL SYSCS_UTIL.SYSCS_EXPORT_QUERY_LOBS_TO_EXTFILE('SELECT * FROM EX_EMP','emp.del', ',' ,'"','UTF-8','pictures.dat'); 0 rows inserted/updated/deleted 6) ij> CALL SYSCS_UTIL.SYSCS_EXPORT_QUERY_LOBS_TO_EXTFILE('SELECT * FROM EX_EMP','emp.del', ',' ,'"','UTF-8','pictures.dat'); ERROR XSDF0: Could not create file emp.del as it already exists. 7) ij> CALL SYSCS_UTIL.SYSCS_EXPORT_QUERY_LOBS_TO_EXTFILE('SELECT * FROM EX_EMP','staff.del', ',' ,'"','UTF-8','pictures.dat'); ERROR XSDF0: Could not create file pictures.dat as it already exists. Thanks Ramin
          Hide
          Kathey Marsden added a comment -

          Hi Ramin,

          The patch looks good except I think it would be good to give import/export its own SQLState starting with XIE instead of reusing the existing store FILE_EXISTS message. Also it would be good to have test cases for writing the lob files.

          Kathey

          Show
          Kathey Marsden added a comment - Hi Ramin, The patch looks good except I think it would be good to give import/export its own SQLState starting with XIE instead of reusing the existing store FILE_EXISTS message. Also it would be good to have test cases for writing the lob files. Kathey
          Hide
          Ramin Moazeni added a comment -

          Hi Kathey

          Thanks for the review. I am attaching another patch
          that I think addresses your comments.

          I appreciate your review.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - Hi Kathey Thanks for the review. I am attaching another patch that I think addresses your comments. I appreciate your review. Thanks Ramin
          Hide
          Bryan Pendleton added a comment -

          I'm comfortable with the general technique; I think it is acceptable for the EXPORT family of procedures to refuse to overwrite existing files.

          A couple comments on the v1 diff:

          1) The "IJ" tests don't seem quite right to me: the tests refer to table DERBY_2925_BOOK but it looks like the actual test table is called derby_2925_lob? Some of the tests seem to be getting table-not-found errors.

          2) I think we should put some effort into making the two messages as clear as possible, both to make it as clear as possible that this is intentional behavior (and not a bug), and to help people understand what they should do when this happens. For example, we don't want people to waste time thinking that there is a file permissions problem here, and trying to re-run the export after fiddling with file permissions.

          Here is a proposed suggestion for the text for the two messages:

          + <msg>
          + <name>XIE0S.S</name>
          + <text>The export operation was not performed, because the specified output file (

          {0}) already exists. Export processing will not overwrite an existing file, even if the process has permissions to write to that file, due to security concerns, and to avoid accidental file damage. Please either change the output file name in the export procedure arguments to specify a file which does not exist, or delete the existing file, then retry the export operation.</text>
          + <arg>fileName</arg>
          + </msg>

          + <msg>
          + <name>XIE0T.S</name>
          + <text>The export operation was not performed, because the specified large object auxiliary file ({0}

          ) already exists. Export processing will not overwrite an existing file, even if the process has permissions to write to that file, due to security concerns, and to avoid accidental file damage. Please either change the large object auxiliary file name in the export procedure arguments to specify a file which does not exist, or delete the existing file, then retry the export operation.</text>
          + <arg>fileName</arg>
          + </msg>

          Show
          Bryan Pendleton added a comment - I'm comfortable with the general technique; I think it is acceptable for the EXPORT family of procedures to refuse to overwrite existing files. A couple comments on the v1 diff: 1) The "IJ" tests don't seem quite right to me: the tests refer to table DERBY_2925_BOOK but it looks like the actual test table is called derby_2925_lob? Some of the tests seem to be getting table-not-found errors. 2) I think we should put some effort into making the two messages as clear as possible, both to make it as clear as possible that this is intentional behavior (and not a bug), and to help people understand what they should do when this happens. For example, we don't want people to waste time thinking that there is a file permissions problem here, and trying to re-run the export after fiddling with file permissions. Here is a proposed suggestion for the text for the two messages: + <msg> + <name>XIE0S.S</name> + <text>The export operation was not performed, because the specified output file ( {0}) already exists. Export processing will not overwrite an existing file, even if the process has permissions to write to that file, due to security concerns, and to avoid accidental file damage. Please either change the output file name in the export procedure arguments to specify a file which does not exist, or delete the existing file, then retry the export operation.</text> + <arg>fileName</arg> + </msg> + <msg> + <name>XIE0T.S</name> + <text>The export operation was not performed, because the specified large object auxiliary file ({0} ) already exists. Export processing will not overwrite an existing file, even if the process has permissions to write to that file, due to security concerns, and to avoid accidental file damage. Please either change the large object auxiliary file name in the export procedure arguments to specify a file which does not exist, or delete the existing file, then retry the export operation.</text> + <arg>fileName</arg> + </msg>
          Hide
          Ramin Moazeni added a comment -

          Thanks Bryan for your review.
          Attachment please find another patch for review.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - Thanks Bryan for your review. Attachment please find another patch for review. Thanks Ramin
          Hide
          Bryan Pendleton added a comment -

          Hi Ramin, thanks for the v2 patch, it looks very good.

          I noticed that part of the patch involves code that will delete the partially-written output file(s) if the EXPORT operation fails. Is that a new behavior of EXPORT? It doesn't seem exactly related to the main issue of DERBY-2925.

          Do you think that the patch would still be valid without the deleteFile portion? Or is that a necessary component of the patch?

          thanks,

          bryan

          Show
          Bryan Pendleton added a comment - Hi Ramin, thanks for the v2 patch, it looks very good. I noticed that part of the patch involves code that will delete the partially-written output file(s) if the EXPORT operation fails. Is that a new behavior of EXPORT? It doesn't seem exactly related to the main issue of DERBY-2925 . Do you think that the patch would still be valid without the deleteFile portion? Or is that a necessary component of the patch? thanks, bryan
          Hide
          Ramin Moazeni added a comment -

          Hi Bryan,

          Thanks for looking at the patch.
          Correct...I added the deleteFile(...) after I got errors running the tests. In this case
          i18n/iepnegativetests_ES.sql was failing and the reason for that was that in executing
          the following call:
          ij> call SYSCS_UTIL.SYSCS_EXPORT_QUERY('select * from iep.t1','t1.dat' , null, null, 'NOSUCHCODESET') ;
          ERROR XIE0I: An IOException occurred while writing data to the file.
          ERROR XJ001: Java exception: 'NOSUCHCODESET: java.io.UnsupportedEncodingException'.

          the output file gets created even though there is an exception and therefore, the rest of the tests were failing because
          the file exists.

          I can remove the deleteFile(...) from the patch but to resolve the test errors, I think another way is to have
          a java stored procedure in in the .sql test to delete the file or simply use different filenames in each
          call to SYSCS_UTIL.SYSCS_EXPORT_... when we know a partially-written file will get created.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - Hi Bryan, Thanks for looking at the patch. Correct...I added the deleteFile(...) after I got errors running the tests. In this case i18n/iepnegativetests_ES.sql was failing and the reason for that was that in executing the following call: ij> call SYSCS_UTIL.SYSCS_EXPORT_QUERY('select * from iep.t1','t1.dat' , null, null, 'NOSUCHCODESET') ; ERROR XIE0I: An IOException occurred while writing data to the file. ERROR XJ001: Java exception: 'NOSUCHCODESET: java.io.UnsupportedEncodingException'. the output file gets created even though there is an exception and therefore, the rest of the tests were failing because the file exists. I can remove the deleteFile(...) from the patch but to resolve the test errors, I think another way is to have a java stored procedure in in the .sql test to delete the file or simply use different filenames in each call to SYSCS_UTIL.SYSCS_EXPORT_... when we know a partially-written file will get created. Thanks Ramin
          Hide
          Bryan Pendleton added a comment -

          Thanks Ramin, that explanation makes complete sense to me.

          I don't have a strong opinion about this deleteFile issue. I think it is nice of the tool to avoid leaving partial files around. The only possible problem I can see is that I think there could be a scenario in which the export fails partway through, and it's not obvious which row of data caused the failure, but by looking at the end of the partial file, the user can see which rows were successfully written just before the failure, and gain a clue about where the failure is occurring.

          So I don't have any particular guidance to give. I think the new deleteFile behavior is useful, but I think it would also be fine to remove that portion of the patch and handle it by changing the tests as you suggested.

          Show
          Bryan Pendleton added a comment - Thanks Ramin, that explanation makes complete sense to me. I don't have a strong opinion about this deleteFile issue. I think it is nice of the tool to avoid leaving partial files around. The only possible problem I can see is that I think there could be a scenario in which the export fails partway through, and it's not obvious which row of data caused the failure, but by looking at the end of the partial file, the user can see which rows were successfully written just before the failure, and gain a clue about where the failure is occurring. So I don't have any particular guidance to give. I think the new deleteFile behavior is useful, but I think it would also be fine to remove that portion of the patch and handle it by changing the tests as you suggested.
          Hide
          Ramin Moazeni added a comment -

          Thanks Bryan and Kathey for your comments.
          I am attaching the v3 patch. I appreciate your review.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - Thanks Bryan and Kathey for your comments. I am attaching the v3 patch. I appreciate your review. Thanks Ramin
          Hide
          Bryan Pendleton added a comment -

          The v3 patch looks great, Ramin! I have no further comments.

          Show
          Bryan Pendleton added a comment - The v3 patch looks great, Ramin! I have no further comments.
          Hide
          Myrna van Lunteren added a comment -

          I think, as we're now requiring users to delete files before attempting to export, that this should have a release note, and it could affect existing applications.

          Please see http://wiki.apache.org/db-derby/ReleaseNoteProcess#head-8bfe22837d50a10f61f410c927336eabc682b62f about writing a release note

          Show
          Myrna van Lunteren added a comment - I think, as we're now requiring users to delete files before attempting to export, that this should have a release note, and it could affect existing applications. Please see http://wiki.apache.org/db-derby/ReleaseNoteProcess#head-8bfe22837d50a10f61f410c927336eabc682b62f about writing a release note
          Hide
          Ramin Moazeni added a comment -

          Hello

          I am attaching the first draft of the release note.
          I appreciate your review.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - Hello I am attaching the first draft of the release note. I appreciate your review. Thanks Ramin
          Hide
          Myrna van Lunteren added a comment -

          I think this change has caused some failures in the tinderbox, see: http://dbtg.thresher.com/derby/test/Daily/jvm1.6/testing/Limited/testSummary-559500.html

          I think those should get resolved before backporting to 10.3.

          Show
          Myrna van Lunteren added a comment - I think this change has caused some failures in the tinderbox, see: http://dbtg.thresher.com/derby/test/Daily/jvm1.6/testing/Limited/testSummary-559500.html I think those should get resolved before backporting to 10.3.
          Hide
          Myrna van Lunteren added a comment -

          Also, I think it would make sense to specify the need for removal of the files in the documentation.

          Show
          Myrna van Lunteren added a comment - Also, I think it would make sense to specify the need for removal of the files in the documentation.
          Hide
          Kathey Marsden added a comment -

          Ramin, do you think the issue is that we now need read permission for the extout directory in the policy file, so we can determine if the file exists? e.g.

          Index: java/testing/org/apache/derbyTesting/functionTests/util/derby_tests.policy
          ===================================================================
          — java/testing/org/apache/derbyTesting/functionTests/util/derby_tests.policy (revision 559646)
          +++ java/testing/org/apache/derbyTesting/functionTests/util/derby_tests.policy (working copy)
          @@ -70,7 +70,7 @@
          // Import/export and other support files from these locations in tests
          permission java.io.FilePermission "$

          {user.dir}${/}extin${/}-", "read";
          permission java.io.FilePermission "${user.dir}

          $

          {/}extinout${/}

          -", "read, write, delete";

          • permission java.io.FilePermission "$ {user.dir}${/}extout${/}-", "write";
            + permission java.io.FilePermission "${user.dir}

            $

            {/}extout${/}

            -", "read,write";
            permission java.io.FilePermission "$

            {user.dir}

            $

            {/}

            extinout", "read,write";

          // These permissions are needed to load the JCE for encryption with Sun and IBM JDK131.
          [C:/svn2/trunk]

          Kathey

          Show
          Kathey Marsden added a comment - Ramin, do you think the issue is that we now need read permission for the extout directory in the policy file, so we can determine if the file exists? e.g. Index: java/testing/org/apache/derbyTesting/functionTests/util/derby_tests.policy =================================================================== — java/testing/org/apache/derbyTesting/functionTests/util/derby_tests.policy (revision 559646) +++ java/testing/org/apache/derbyTesting/functionTests/util/derby_tests.policy (working copy) @@ -70,7 +70,7 @@ // Import/export and other support files from these locations in tests permission java.io.FilePermission "$ {user.dir}${/}extin${/}-", "read"; permission java.io.FilePermission "${user.dir} $ {/}extinout${/} -", "read, write, delete"; permission java.io.FilePermission "$ {user.dir}${/}extout${/}-", "write"; + permission java.io.FilePermission "${user.dir} $ {/}extout${/} -", "read,write"; permission java.io.FilePermission "$ {user.dir} $ {/} extinout", "read,write"; // These permissions are needed to load the JCE for encryption with Sun and IBM JDK131. [C:/svn2/trunk] Kathey
          Hide
          Ramin Moazeni added a comment -

          Attaching the v4 patch which resolve some of the tests errors.
          Please review.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - Attaching the v4 patch which resolve some of the tests errors. Please review. Thanks Ramin
          Hide
          Ramin Moazeni added a comment -

          Hello

          I am attaching a possible fix for the AccessControl exception that is being thrown.
          The fix contains changing the order for the tests that are being executed in
          suites/AllPackages.java. I will log a separate issue for fixing possible
          issues with the tests that might change the policy and have problem restoring
          the policy.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - Hello I am attaching a possible fix for the AccessControl exception that is being thrown. The fix contains changing the order for the tests that are being executed in suites/AllPackages.java. I will log a separate issue for fixing possible issues with the tests that might change the policy and have problem restoring the policy. Thanks Ramin
          Hide
          Kathey Marsden added a comment -

          Running suites.All with the patch I see these failures: Almost as though the permissions problem has moved.

          3) testIllegalOps(org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest)junit.framework.ComparisonFailure:
          Unexpected SQL state. expected:<42Z7...> but was:<XJ00...>
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:624)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:659)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:673)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:854)
          at org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest.testIllegalOps(XMLTypeAndOpsTest.java:352)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:95)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          Caused by: java.sql.SQLException: Java exception: 'Access denied (java.io.FilePermission xmlexport.del read): java.secur
          ity.AccessControlException'.
          at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
          at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:88)
          at org.apache.derby.impl.jdbc.Util.javaException(Util.java:245)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:403)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:398)
          at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:346)
          at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:1572)
          at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1293)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1652)
          at org.apache.derby.impl.jdbc.EmbedCallableStatement.executeStatement(EmbedCallableStatement.java:116)
          at org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(EmbedPreparedStatement.java:1304)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:849)
          ... 34 more
          Caused by: java.security.AccessControlException: Access denied (java.io.FilePermission xmlexport.del read)
          at java.security.AccessController.checkPermission(AccessController.java:104)
          at java.lang.SecurityManager.checkPermission(SecurityManager.java:547)
          at java.lang.SecurityManager.checkRead(SecurityManager.java:886)
          at java.io.File.exists(File.java:726)
          at org.apache.derby.iapi.util.PrivilegedFileOps$1.run(PrivilegedFileOps.java:60)
          at java.security.AccessController.doPrivileged(AccessController.java:242)
          at org.apache.derby.iapi.util.PrivilegedFileOps.exists(PrivilegedFileOps.java:57)
          at org.apache.derby.impl.load.Export.dataFileExists(Export.java:146)
          at org.apache.derby.impl.load.Export.doExport(Export.java:57)
          at org.apache.derby.impl.load.Export.exportTable(Export.java:172)
          at org.apache.derby.catalog.SystemProcedures.SYSCS_EXPORT_TABLE(SystemProcedures.java:1128)
          at org.apache.derby.exe.ac592dcde3x0114x19dfx7bc8xffffa650e7100.g0(Unknown Source)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at org.apache.derby.impl.services.reflect.ReflectMethod.invoke(ReflectMethod.java:46)
          at org.apache.derby.impl.sql.execute.CallStatementResultSet.open(CallStatementResultSet.java:57)
          at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:370)
          at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1203)
          ... 38 more
          4) testIllegalOps(org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest)junit.framework.ComparisonFailure:
          Unexpected SQL state. expected:<42Z7...> but was:<XJ00...>
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:624)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:659)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:673)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:854)
          at org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest.testIllegalOps(XMLTypeAndOpsTest.java:352)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:95)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          Caused by: java.sql.SQLException: Java exception: 'Access denied (java.io.FilePermission xmlexport.del read): java.secur
          ity.AccessControlException'.
          at org.apache.derby.client.am.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:46)
          at org.apache.derby.client.am.SqlException.getSQLException(SqlException.java:362)
          at org.apache.derby.client.am.SqlException.getSQLException(SqlException.java:371)
          at org.apache.derby.client.am.PreparedStatement.execute(PreparedStatement.java:1572)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:849)
          ... 43 more
          Caused by: org.apache.derby.client.am.SqlException: Java exception: 'Access denied (java.io.FilePermission xmlexport.del
          read): java.security.AccessControlException'.
          at org.apache.derby.client.am.SqlException.<init>(SqlException.java:290)
          at org.apache.derby.client.am.SqlException.<init>(SqlException.java:264)
          at org.apache.derby.client.am.Statement.completeExecute(Statement.java:1498)
          at org.apache.derby.client.net.NetStatementReply.parseEXCSQLSTTreply(NetStatementReply.java:304)
          at org.apache.derby.client.net.NetStatementReply.readExecuteCall(NetStatementReply.java:105)
          at org.apache.derby.client.net.StatementReply.readExecuteCall(StatementReply.java:75)
          at org.apache.derby.client.net.NetStatement.readExecuteCall_(NetStatement.java:176)
          at org.apache.derby.client.am.Statement.readExecuteCall(Statement.java:1464)
          at org.apache.derby.client.am.PreparedStatement.flowExecute(PreparedStatement.java:2158)
          at org.apache.derby.client.am.PreparedStatement.executeX(PreparedStatement.java:1578)
          at org.apache.derby.client.am.PreparedStatement.execute(PreparedStatement.java:1563)
          ... 44 more

          Show
          Kathey Marsden added a comment - Running suites.All with the patch I see these failures: Almost as though the permissions problem has moved. 3) testIllegalOps(org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest)junit.framework.ComparisonFailure: Unexpected SQL state. expected:<42Z7...> but was:<XJ00...> at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:624) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:659) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:673) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:854) at org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest.testIllegalOps(XMLTypeAndOpsTest.java:352) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:95) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) Caused by: java.sql.SQLException: Java exception: 'Access denied (java.io.FilePermission xmlexport.del read): java.secur ity.AccessControlException'. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:88) at org.apache.derby.impl.jdbc.Util.javaException(Util.java:245) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:403) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:398) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:346) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:1572) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1293) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1652) at org.apache.derby.impl.jdbc.EmbedCallableStatement.executeStatement(EmbedCallableStatement.java:116) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(EmbedPreparedStatement.java:1304) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:849) ... 34 more Caused by: java.security.AccessControlException: Access denied (java.io.FilePermission xmlexport.del read) at java.security.AccessController.checkPermission(AccessController.java:104) at java.lang.SecurityManager.checkPermission(SecurityManager.java:547) at java.lang.SecurityManager.checkRead(SecurityManager.java:886) at java.io.File.exists(File.java:726) at org.apache.derby.iapi.util.PrivilegedFileOps$1.run(PrivilegedFileOps.java:60) at java.security.AccessController.doPrivileged(AccessController.java:242) at org.apache.derby.iapi.util.PrivilegedFileOps.exists(PrivilegedFileOps.java:57) at org.apache.derby.impl.load.Export.dataFileExists(Export.java:146) at org.apache.derby.impl.load.Export.doExport(Export.java:57) at org.apache.derby.impl.load.Export.exportTable(Export.java:172) at org.apache.derby.catalog.SystemProcedures.SYSCS_EXPORT_TABLE(SystemProcedures.java:1128) at org.apache.derby.exe.ac592dcde3x0114x19dfx7bc8xffffa650e7100.g0(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derby.impl.services.reflect.ReflectMethod.invoke(ReflectMethod.java:46) at org.apache.derby.impl.sql.execute.CallStatementResultSet.open(CallStatementResultSet.java:57) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:370) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1203) ... 38 more 4) testIllegalOps(org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest)junit.framework.ComparisonFailure: Unexpected SQL state. expected:<42Z7...> but was:<XJ00...> at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:624) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:659) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertSQLState(BaseJDBCTestCase.java:673) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:854) at org.apache.derbyTesting.functionTests.tests.lang.XMLTypeAndOpsTest.testIllegalOps(XMLTypeAndOpsTest.java:352) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:95) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) Caused by: java.sql.SQLException: Java exception: 'Access denied (java.io.FilePermission xmlexport.del read): java.secur ity.AccessControlException'. at org.apache.derby.client.am.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:46) at org.apache.derby.client.am.SqlException.getSQLException(SqlException.java:362) at org.apache.derby.client.am.SqlException.getSQLException(SqlException.java:371) at org.apache.derby.client.am.PreparedStatement.execute(PreparedStatement.java:1572) at org.apache.derbyTesting.junit.BaseJDBCTestCase.assertStatementError(BaseJDBCTestCase.java:849) ... 43 more Caused by: org.apache.derby.client.am.SqlException: Java exception: 'Access denied (java.io.FilePermission xmlexport.del read): java.security.AccessControlException'. at org.apache.derby.client.am.SqlException.<init>(SqlException.java:290) at org.apache.derby.client.am.SqlException.<init>(SqlException.java:264) at org.apache.derby.client.am.Statement.completeExecute(Statement.java:1498) at org.apache.derby.client.net.NetStatementReply.parseEXCSQLSTTreply(NetStatementReply.java:304) at org.apache.derby.client.net.NetStatementReply.readExecuteCall(NetStatementReply.java:105) at org.apache.derby.client.net.StatementReply.readExecuteCall(StatementReply.java:75) at org.apache.derby.client.net.NetStatement.readExecuteCall_(NetStatement.java:176) at org.apache.derby.client.am.Statement.readExecuteCall(Statement.java:1464) at org.apache.derby.client.am.PreparedStatement.flowExecute(PreparedStatement.java:2158) at org.apache.derby.client.am.PreparedStatement.executeX(PreparedStatement.java:1578) at org.apache.derby.client.am.PreparedStatement.execute(PreparedStatement.java:1563) ... 44 more
          Hide
          Ramin Moazeni added a comment -

          Hi Kathey

          Thanks for posting the error messages.
          I am attaching a new patch which hopefully resolve this issue.

          I did not get this error message since I didn't have xalan.jar in my classpath
          and the test was being skipped.

          Thanks
          Ramin

          Show
          Ramin Moazeni added a comment - Hi Kathey Thanks for posting the error messages. I am attaching a new patch which hopefully resolve this issue. I did not get this error message since I didn't have xalan.jar in my classpath and the test was being skipped. Thanks Ramin
          Hide
          Ramin Moazeni added a comment -

          To merge to 10.3 branch use the following commends:
          svn merge -r 559168:559169 https://svn.apache.org/repos/asf/db/derby/code/trunk
          svn merge -r 560423:560424 https://svn.apache.org/repos/asf/db/derby/code/trunk
          svn merge -r 561545:561546 https://svn.apache.org/repos/asf/db/derby/code/trunk

          Show
          Ramin Moazeni added a comment - To merge to 10.3 branch use the following commends: svn merge -r 559168:559169 https://svn.apache.org/repos/asf/db/derby/code/trunk svn merge -r 560423:560424 https://svn.apache.org/repos/asf/db/derby/code/trunk svn merge -r 561545:561546 https://svn.apache.org/repos/asf/db/derby/code/trunk
          Hide
          Kathey Marsden added a comment -

          resolved in trunk and 10.3

          Show
          Kathey Marsden added a comment - resolved in trunk and 10.3
          Hide
          Rick Hillegas added a comment -

          Re-attaching release note as releaseNote.html since this is the file name which the release notes generator looks for.

          Show
          Rick Hillegas added a comment - Re-attaching release note as releaseNote.html since this is the file name which the release notes generator looks for.
          Hide
          Rick Hillegas added a comment -

          Re-opening this issue to handle another case.

          Show
          Rick Hillegas added a comment - Re-opening this issue to handle another case.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-2925-07-aa-fileUrl.diff. This patch prevents users from subverting the file existence check by phrasing the filename as an url. The tests passed cleanly for me.

          The export procedure uses the user-supplied export file name twice:

          1) To check whether the file exists

          2) To open the file so that export can dump a table into the file

          For the second use, Derby checks to see whether the user-supplied name is actually an url rather than a plain file name. If the user-supplied name is an url, Derby transforms it into a plain file name. This transformation is not performed when checking for the file's existence. As a result, it is still possible to use export to overwrite an existing file by prefixing the file name with the string "file:". This will be an illegal file name for the purposes of (1) and therefore not abort the export before step (2).

          This patch simply performs the same transformation in steps (1) and (2).

          Touches the following files:

          ------------

          M java/engine/org/apache/derby/iapi/services/io/FileUtil.java
          M java/engine/org/apache/derby/impl/load/Export.java
          M java/engine/org/apache/derby/impl/load/ExportWriteData.java

          Factor out the file name transformation into FileUtil so that it can be used by both steps (1) and (2).

          ------------

          M java/engine/org/apache/derby/impl/io/CPFile.java

          Removed an unneeded import.

          ------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/tools/ImportExportBinaryDataTest.java

          Added a test case for this scenario.

          Show
          Rick Hillegas added a comment - Attaching derby-2925-07-aa-fileUrl.diff. This patch prevents users from subverting the file existence check by phrasing the filename as an url. The tests passed cleanly for me. The export procedure uses the user-supplied export file name twice: 1) To check whether the file exists 2) To open the file so that export can dump a table into the file For the second use, Derby checks to see whether the user-supplied name is actually an url rather than a plain file name. If the user-supplied name is an url, Derby transforms it into a plain file name. This transformation is not performed when checking for the file's existence. As a result, it is still possible to use export to overwrite an existing file by prefixing the file name with the string "file:". This will be an illegal file name for the purposes of (1) and therefore not abort the export before step (2). This patch simply performs the same transformation in steps (1) and (2). Touches the following files: ------------ M java/engine/org/apache/derby/iapi/services/io/FileUtil.java M java/engine/org/apache/derby/impl/load/Export.java M java/engine/org/apache/derby/impl/load/ExportWriteData.java Factor out the file name transformation into FileUtil so that it can be used by both steps (1) and (2). ------------ M java/engine/org/apache/derby/impl/io/CPFile.java Removed an unneeded import. ------------ M java/testing/org/apache/derbyTesting/functionTests/tests/tools/ImportExportBinaryDataTest.java Added a test case for this scenario.
          Hide
          Bryan Pendleton added a comment -

          Your patch looks good to me, thanks for tracking this down.

          I'm a little surprised that in your test, you were able to do

          "file:" + fileName

          rather than

          "file://" + fileName

          No other comments. +1 to commit.

          Show
          Bryan Pendleton added a comment - Your patch looks good to me, thanks for tracking this down. I'm a little surprised that in your test, you were able to do "file:" + fileName rather than "file://" + fileName No other comments. +1 to commit.
          Hide
          Rick Hillegas added a comment -

          Thanks for the quick review, Bryan. Committed patch at subversion revision 957171.

          Show
          Rick Hillegas added a comment - Thanks for the quick review, Bryan. Committed patch at subversion revision 957171.
          Hide
          Rick Hillegas added a comment -

          Ported 957171 from trunk to 10.6 branch at subversion revision 957278.

          Show
          Rick Hillegas added a comment - Ported 957171 from trunk to 10.6 branch at subversion revision 957278.

            People

            • Assignee:
              Ramin Moazeni
              Reporter:
              Kathey Marsden
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development