Derby
  1. Derby
  2. DERBY-5968

A failed connection attempt may nevertheless manage to boot the database.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 10.10.1.1
    • Fix Version/s: None
    • Component/s: Services
    • Urgency:
      Normal
    • Issue & fix info:
      Repro attached
    • Bug behavior facts:
      Security

      Description

      A connection attempt may fail but still manage to boot the database. If possible, we should try to bring down the database after the connection failure.

      1) Here is an example of this problem: If you use bad credentials to connect to a database, you will fail to get a connection. That's correct behavior. However, Derby must boot the database in order to discover its authentication mechanism. The database remains booted after this check.

      2) There are other examples of this problem. For instance, if a user with good credentials tries to change the boot password on an encrypted database, then the connection attempt will fail, but the database will remain booted. This can cause silent failures on later attempts to re-encrypt or un-encrypt a database after the low-privilege or nonexistent user manages to boot the database. The connection attempt will succeed, leading the DBO to think that the re-encryption worked. But actually re-encryption did not take place because the database was already booted.

      I regard this silent failure as a security vulnerability.

      The following script shows problem (1):

      connect 'jdbc:derby:db;create=true;user=test_dbo';

      call syscs_util.syscs_create_user( 'test_dbo', 'test_dbopassword' );

      – shutdown the database
      connect 'jdbc:derby:db;shutdown=true';

      – need credentials in order to get a connection
      connect 'jdbc:derby:db';
      connect 'jdbc:derby:db;user=test_dbo;password=test_dbopassword';
      select count from sys.systables;

      – shutdown the database
      connect 'jdbc:derby:db;shutdown=true;user=test_dbo;password=test_dbopassword';

      – try to shutdown the database again. since it is already shutdown, you will get a message saying it can't be found.
      connect 'jdbc:derby:db;shutdown=true;user=test_dbo;password=test_dbopassword';

      – now try to boot with bad credentials. you don't get a connection but the database boots.
      connect 'jdbc:derby:db;user=alice;password=alicepassword;bootPassword=foobarwibblewombat';
      select count from sys.systables;

      – the shutdown command succeeds because alice managed to boot the database
      – even though she isn't a legal user.
      connect 'jdbc:derby:db;shutdown=true;user=test_dbo;password=test_dbopassword';

        Issue Links

          Activity

          Hide
          Rick Hillegas added a comment -

          Here is a script showing how this bug causes unencryption to silently fail:

          connect 'jdbc:derby:db;create=true;user=test_dbo;dataEncryption=true;bootPassword=foobarwibblewombat';

          call syscs_util.syscs_create_user( 'test_dbo', 'test_dbopassword' );
          call syscs_util.syscs_create_user( 'fred', 'fredpassword' );

          – shutdown the database
          connect 'jdbc:derby:db;shutdown=true';

          – only the dbo can unencrypt the database
          connect 'jdbc:derby:db;user=fred;password=fredpassword;bootPassword=foobarwibblewombat;decryptDatabase=true';

          – although the connection failed, the database is now booted.
          – the following attempt to decrypt the database appears to work
          – but actually fails.
          connect 'jdbc:derby:db;user=test_dbo;password=test_dbopassword;bootPassword=foobarwibblewombat;decryptDatabase=true';

          – really shutdown the database
          connect 'jdbc:derby:db;shutdown=true;user=test_dbo;password=test_dbopassword';

          – this fails because the unencryption silently failed. a boot password is still required.
          connect 'jdbc:derby:db;user=test_dbo;password=test_dbopassword';

          Show
          Rick Hillegas added a comment - Here is a script showing how this bug causes unencryption to silently fail: connect 'jdbc:derby:db;create=true;user=test_dbo;dataEncryption=true;bootPassword=foobarwibblewombat'; call syscs_util.syscs_create_user( 'test_dbo', 'test_dbopassword' ); call syscs_util.syscs_create_user( 'fred', 'fredpassword' ); – shutdown the database connect 'jdbc:derby:db;shutdown=true'; – only the dbo can unencrypt the database connect 'jdbc:derby:db;user=fred;password=fredpassword;bootPassword=foobarwibblewombat;decryptDatabase=true'; – although the connection failed, the database is now booted. – the following attempt to decrypt the database appears to work – but actually fails. connect 'jdbc:derby:db;user=test_dbo;password=test_dbopassword;bootPassword=foobarwibblewombat;decryptDatabase=true'; – really shutdown the database connect 'jdbc:derby:db;shutdown=true;user=test_dbo;password=test_dbopassword'; – this fails because the unencryption silently failed. a boot password is still required. connect 'jdbc:derby:db;user=test_dbo;password=test_dbopassword';
          Hide
          Rick Hillegas added a comment -

          Attaching derby-5968-01-aa-shutdownDatabaseIfBootingConnectionFails.diff. This patch shuts down the database if it was booted as a side-effect of a failed connection attempt.

          I still need to do the following:

          1) Add a test case to verify this behavior.

          2) Run the full regression tests. I suspect that this change may affect several tests.

          Touches the following file:

          M java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java

          Show
          Rick Hillegas added a comment - Attaching derby-5968-01-aa-shutdownDatabaseIfBootingConnectionFails.diff. This patch shuts down the database if it was booted as a side-effect of a failed connection attempt. I still need to do the following: 1) Add a test case to verify this behavior. 2) Run the full regression tests. I suspect that this change may affect several tests. Touches the following file: M java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java
          Hide
          Rick Hillegas added a comment -

          Attaching derby-5968-01-ab-shutdownDatabaseIfBootingConnectionFails.diff. This second rev of the patch adds a test to verify the new behavior and fixes some failed tests. I will re-run regression tests.

          Touches the following files:

          ----------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/jdbcapi/DSCreateShutdownDBTest.java

          Test case to verify new behavior.

          ----------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/jdbcapi/DboPowersTest.java
          M java/testing/org/apache/derbyTesting/junit/BaseJDBCTestCase.java

          Added an overload for BaseJDBCTestCase.assertSQLState() which lets you specify a set of possible SQLStates which are considered OK. This was needed by DboPowersTest because it needs to repeatedly return to an initial position of an unbooted database but it doesn't care whether the database is actually up or down when the test returns to initial position.

          Show
          Rick Hillegas added a comment - Attaching derby-5968-01-ab-shutdownDatabaseIfBootingConnectionFails.diff. This second rev of the patch adds a test to verify the new behavior and fixes some failed tests. I will re-run regression tests. Touches the following files: ---------------- M java/testing/org/apache/derbyTesting/functionTests/tests/jdbcapi/DSCreateShutdownDBTest.java Test case to verify new behavior. ---------------- M java/testing/org/apache/derbyTesting/functionTests/tests/jdbcapi/DboPowersTest.java M java/testing/org/apache/derbyTesting/junit/BaseJDBCTestCase.java Added an overload for BaseJDBCTestCase.assertSQLState() which lets you specify a set of possible SQLStates which are considered OK. This was needed by DboPowersTest because it needs to repeatedly return to an initial position of an unbooted database but it doesn't care whether the database is actually up or down when the test returns to initial position.
          Hide
          Rick Hillegas added a comment -

          Attaching rev 3 of the patch: derby-5968-01-ac-shutdownDatabaseIfBootingConnectionFails.diff. The previous patch tripped a problem in Derby5937SlaveShutdownTest. I will run regression tests on this patch if people think that its approach is correct.

          I believe this is what was happening in Derby5937SlaveShutdownTest:

          o The test loops, trying to connect to the master in order to start replication. For a couple iterations, the connection attempts fail because the slave has not come up yet.

          o With each failed connection attempt, the master database boots and then shuts down.

          o Shutting down the master database moves its log instant forward. This breaks agreement between the master and slave databases.

          o When the slave does come up, it notices that the log instant of the master has moved forward. Replication fails because the two databases are not in agreement.

          Here are two approaches to this problem:

          1) Treat this as a defect in the test. Figure out some way to delay booting the master until the slave has come up.

          2) Treat this as a coding pattern which applications may already rely on. Put some defensive code in EmbedConnection so that we don't bring down the master database if the slave has not come up yet.

          The third patch implements approach (2). However, I am not convinced that we should support this application coding pattern. In particular, I think that the failure to bring down the master database in this case will still leave a security hole. I would like other people's opinions about whether we should treat the failure of Derby5937SlaveShutdownTest as a defect in the test rather than a product bug.

          Thanks in advance for your opinions,
          -Rick

          Touches the following additional files:

          ----------------

          M java/build/org/apache/derbyBuild/MessageBundleTest.java
          M java/shared/org/apache/derby/shared/common/reference/SQLState.java

          Factor out a SQLState which does not bring down the master database.

          ----------------

          M java/testing/org/apache/derbyTesting/functionTests/tests/replicationTests/Derby5937SlaveShutdownTest.java

          Put a little more debug instrumentation in this test.

          Show
          Rick Hillegas added a comment - Attaching rev 3 of the patch: derby-5968-01-ac-shutdownDatabaseIfBootingConnectionFails.diff. The previous patch tripped a problem in Derby5937SlaveShutdownTest. I will run regression tests on this patch if people think that its approach is correct. I believe this is what was happening in Derby5937SlaveShutdownTest: o The test loops, trying to connect to the master in order to start replication. For a couple iterations, the connection attempts fail because the slave has not come up yet. o With each failed connection attempt, the master database boots and then shuts down. o Shutting down the master database moves its log instant forward. This breaks agreement between the master and slave databases. o When the slave does come up, it notices that the log instant of the master has moved forward. Replication fails because the two databases are not in agreement. Here are two approaches to this problem: 1) Treat this as a defect in the test. Figure out some way to delay booting the master until the slave has come up. 2) Treat this as a coding pattern which applications may already rely on. Put some defensive code in EmbedConnection so that we don't bring down the master database if the slave has not come up yet. The third patch implements approach (2). However, I am not convinced that we should support this application coding pattern. In particular, I think that the failure to bring down the master database in this case will still leave a security hole. I would like other people's opinions about whether we should treat the failure of Derby5937SlaveShutdownTest as a defect in the test rather than a product bug. Thanks in advance for your opinions, -Rick Touches the following additional files: ---------------- M java/build/org/apache/derbyBuild/MessageBundleTest.java M java/shared/org/apache/derby/shared/common/reference/SQLState.java Factor out a SQLState which does not bring down the master database. ---------------- M java/testing/org/apache/derbyTesting/functionTests/tests/replicationTests/Derby5937SlaveShutdownTest.java Put a little more debug instrumentation in this test.
          Hide
          Knut Anders Hatlen added a comment -

          Since the startSlave command doesn't return until the slave is connected to the master, I don't think we have any way with the current API to tell that the slave is up and ready to accept connections from the master. We could of course make the test sleep a little while before attempting to start the master, but that doesn't sound very robust.

          I think all the replication tests use this approach when they start the master. See for example the ReplicationRun.startMaster_direct() method. However, most of the other replication tests run the master in a separate process so that the timing could be different, which might explain why you only see this in one test.

          Show
          Knut Anders Hatlen added a comment - Since the startSlave command doesn't return until the slave is connected to the master, I don't think we have any way with the current API to tell that the slave is up and ready to accept connections from the master. We could of course make the test sleep a little while before attempting to start the master, but that doesn't sound very robust. I think all the replication tests use this approach when they start the master. See for example the ReplicationRun.startMaster_direct() method. However, most of the other replication tests run the master in a separate process so that the timing could be different, which might explain why you only see this in one test.
          Hide
          Rick Hillegas added a comment -

          Tests passed cleanly for me on derby-5968-01-ac-shutdownDatabaseIfBootingConnectionFails.diff except for two known heisenbugs. I tried running the failed tests standalone 10 times apiece but the errors did not recur. The heisenbugs are:

          DERBY-5941
          DERBY-5942 - For this one, the stack trace is slightly different, perhaps because of the patch which Kristian committed for DERBY-5942.

          Here is the failure report:

          There were 2 failures:
          1) testPingWithWrongHost(org.apache.derbyTesting.functionTests.tests.derbynet.NetworkServerControlClientCommandTest)junit.framework.AssertionFailedError: Could not find expectedString:Unable to find host in output:<STDOUT>Tue Nov 06 13:48:02 PST 2012 : Could not connect to Derby Network Server on host nothere.invalid, port 1527: Operation timed out
          <END STDOUT>
          <STDERR><END STDERR>

          at org.apache.derbyTesting.junit.BaseTestCase.assertExecJavaCmdAsExpected(BaseTestCase.java:524)
          at org.apache.derbyTesting.functionTests.tests.derbynet.NetworkServerControlClientCommandTest.assertFailedPing(NetworkServerControlClientCommandTest.java:148)
          at org.apache.derbyTesting.functionTests.tests.derbynet.NetworkServerControlClientCommandTest.testPingWithWrongHost(NetworkServerControlClientCommandTest.java:113)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:117)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:424)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:441)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          at junit.extensions.TestSetup.run(TestSetup.java:25)
          at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          at junit.extensions.TestSetup.run(TestSetup.java:25)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          at junit.extensions.TestSetup.run(TestSetup.java:25)
          2) testInvalidLDAPServerConnectionError(org.apache.derbyTesting.functionTests.tests.jdbcapi.InvalidLDAPServerAuthenticationTest)junit.framework.AssertionFailedError
          at org.apache.derbyTesting.functionTests.tests.jdbcapi.InvalidLDAPServerAuthenticationTest.testInvalidLDAPServerConnectionError(InvalidLDAPServerAuthenticationTest.java:122)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:117)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:424)
          at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:441)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          at junit.extensions.TestSetup.run(TestSetup.java:25)
          at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          at junit.extensions.TestSetup.run(TestSetup.java:25)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
          at junit.extensions.TestSetup.run(TestSetup.java:25)

          FAILURES!!!
          Tests run: 13716, Failures: 2, Errors: 0

          Show
          Rick Hillegas added a comment - Tests passed cleanly for me on derby-5968-01-ac-shutdownDatabaseIfBootingConnectionFails.diff except for two known heisenbugs. I tried running the failed tests standalone 10 times apiece but the errors did not recur. The heisenbugs are: DERBY-5941 DERBY-5942 - For this one, the stack trace is slightly different, perhaps because of the patch which Kristian committed for DERBY-5942 . Here is the failure report: There were 2 failures: 1) testPingWithWrongHost(org.apache.derbyTesting.functionTests.tests.derbynet.NetworkServerControlClientCommandTest)junit.framework.AssertionFailedError: Could not find expectedString:Unable to find host in output:<STDOUT>Tue Nov 06 13:48:02 PST 2012 : Could not connect to Derby Network Server on host nothere.invalid, port 1527: Operation timed out <END STDOUT> <STDERR><END STDERR> at org.apache.derbyTesting.junit.BaseTestCase.assertExecJavaCmdAsExpected(BaseTestCase.java:524) at org.apache.derbyTesting.functionTests.tests.derbynet.NetworkServerControlClientCommandTest.assertFailedPing(NetworkServerControlClientCommandTest.java:148) at org.apache.derbyTesting.functionTests.tests.derbynet.NetworkServerControlClientCommandTest.testPingWithWrongHost(NetworkServerControlClientCommandTest.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:117) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:424) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:441) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) 2) testInvalidLDAPServerConnectionError(org.apache.derbyTesting.functionTests.tests.jdbcapi.InvalidLDAPServerAuthenticationTest)junit.framework.AssertionFailedError at org.apache.derbyTesting.functionTests.tests.jdbcapi.InvalidLDAPServerAuthenticationTest.testInvalidLDAPServerConnectionError(InvalidLDAPServerAuthenticationTest.java:122) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:117) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBareOverridable(BaseJDBCTestCase.java:424) at org.apache.derbyTesting.junit.BaseJDBCTestCase.runBare(BaseJDBCTestCase.java:441) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24) at junit.extensions.TestSetup$1.protect(TestSetup.java:21) at junit.extensions.TestSetup.run(TestSetup.java:25) FAILURES!!! Tests run: 13716, Failures: 2, Errors: 0
          Hide
          Rick Hillegas added a comment -

          I have linked this issue to DERBY-2407 which also addresses the topic of accidentally booted databases. Discussion there notes that Derby would consume even more CPU if it deliberately brought down the database after an accidental boot and that, in turn, would make a denial-of-service attack consume more CPU. I think that there is a tradeoff between that denial-of-service-attack and the current denial-of-service attack of consuming memory by leaving an accidentally booted database up and running. And I don't think that either attack is likely or worth worrying about.

          More seriously, DERBY-2407 discusses race conditions which could arise when more than one user tries to boot the database by connecting to it. We would want to make sure that shutting down after an accidental boot didn't orphan other users who are racing us to connect. DERBY-4447 might give us the tools to do this.

          I am halting work on this issue. Before proceeding further, I think we would want to:

          1) Implement DERBY-4447 so that we have a clean mechanism for handling connection races.

          2) Clean up the replication api so that we can remove the hack in derby-5968-01-ac-shutdownDatabaseIfBootingConnectionFails.diff.

          Show
          Rick Hillegas added a comment - I have linked this issue to DERBY-2407 which also addresses the topic of accidentally booted databases. Discussion there notes that Derby would consume even more CPU if it deliberately brought down the database after an accidental boot and that, in turn, would make a denial-of-service attack consume more CPU. I think that there is a tradeoff between that denial-of-service-attack and the current denial-of-service attack of consuming memory by leaving an accidentally booted database up and running. And I don't think that either attack is likely or worth worrying about. More seriously, DERBY-2407 discusses race conditions which could arise when more than one user tries to boot the database by connecting to it. We would want to make sure that shutting down after an accidental boot didn't orphan other users who are racing us to connect. DERBY-4447 might give us the tools to do this. I am halting work on this issue. Before proceeding further, I think we would want to: 1) Implement DERBY-4447 so that we have a clean mechanism for handling connection races. 2) Clean up the replication api so that we can remove the hack in derby-5968-01-ac-shutdownDatabaseIfBootingConnectionFails.diff.

            People

            • Assignee:
              Unassigned
              Reporter:
              Rick Hillegas
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development