Derby
  1. Derby
  2. DERBY-3541

quit; on ij with authentication enabled does not shutdown the derby modules that are loaded

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 10.4.1.3
    • Fix Version/s: None
    • Component/s: Tools
    • Urgency:
      Normal
    • Bug behavior facts:
      Security
    1. PrintStackTrace.diff
      0.5 kB
      V.Narayanan
    2. WithoutAuthentication_StackTrace.txt
      4 kB
      V.Narayanan
    3. derby.properties
      0.1 kB
      V.Narayanan

      Issue Links

        Activity

        Hide
        V.Narayanan added a comment -

        I modified the derby codebase to print a stack trace (new Throwable().printStackTrace() in
        the TopService#shutdown() method (pls find patch attached.)

        I then tried getting a connection with authentication enabled (pls find derby.properties
        attached) with the following connection url

        ij> connect 'jdbc:derby:replicationdb2;create=true;user=oystein;password=pass';
        ij> quit;
        vn@vn-laptop:~/work/workspaces/PrintStackTrace/trials$

        There was no stack trace printed.

        I then tried a connection without authentication enabled

        There was a stack trace printed (pls find attached stack trace)

        ij> connect 'jdbc:derby:replicationdb1;create=true';
        ij> quit;
        java.lang.Throwable
        at org.apache.derby.impl.services.monitor.TopService.shutdown(TopService.java:327)
        at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:237)
        at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:203)
        at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:231)
        at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:207)
        at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416)
        at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:249)
        at org.apache.derby.impl.tools.ij.Main.go(Main.java:215)
        at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:181)
        at org.apache.derby.impl.tools.ij.Main.main(Main.java:73)
        at org.apache.derby.tools.ij.main(ij.java:59)

        ............... and so on (attached is the full stack trace)

        Not shutting down the modules when authentication is enabled I believe is not correct.

        Show
        V.Narayanan added a comment - I modified the derby codebase to print a stack trace (new Throwable().printStackTrace() in the TopService#shutdown() method (pls find patch attached.) I then tried getting a connection with authentication enabled (pls find derby.properties attached) with the following connection url ij> connect 'jdbc:derby:replicationdb2;create=true;user=oystein;password=pass'; ij> quit; vn@vn-laptop:~/work/workspaces/PrintStackTrace/trials$ There was no stack trace printed. I then tried a connection without authentication enabled There was a stack trace printed (pls find attached stack trace) ij> connect 'jdbc:derby:replicationdb1;create=true'; ij> quit; java.lang.Throwable at org.apache.derby.impl.services.monitor.TopService.shutdown(TopService.java:327) at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:237) at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:203) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:231) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:207) at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416) at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:249) at org.apache.derby.impl.tools.ij.Main.go(Main.java:215) at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:181) at org.apache.derby.impl.tools.ij.Main.main(Main.java:73) at org.apache.derby.tools.ij.main(ij.java:59) ............... and so on (attached is the full stack trace) Not shutting down the modules when authentication is enabled I believe is not correct.
        Hide
        V.Narayanan added a comment -

        Derby-3447 relies on the stop() on the MasterController module
        being called in order to shutdown the log shipper thread. Since
        stop() is not called in MasterController upon a quit; in ij when authentication
        is enabled it results in the log shipper thread being active and cause
        a hang after quit; is called.

        Show
        V.Narayanan added a comment - Derby-3447 relies on the stop() on the MasterController module being called in order to shutdown the log shipper thread. Since stop() is not called in MasterController upon a quit; in ij when authentication is enabled it results in the log shipper thread being active and cause a hang after quit; is called.
        Hide
        V.Narayanan added a comment -

        Upon calling quit in ij, ij tries to shutdown the connection. The code relevant to this
        is found here

        org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416)

        here we do the following

        if (d!=null) { // do we have a driver running? shutdown on exit.
        try

        { DriverManager.getConnection("jdbc:derby:;shutdown=true"); }

        catch (SQLException e)

        { // ignore the errors, they are expected. }

        }

        now with authentication enabled trying to take a connection to shutdown like done in
        the above code will result in an exception

        ij> connect 'jdbc:derby:;shutdown=true';
        ERROR 08004: Connection authentication failure occurred. Reason: Invalid authentication..

        This is the reason why upon a quit; in ij with authentication enabled and in the
        embedded mode the modules are never shutdown.

        Show
        V.Narayanan added a comment - Upon calling quit in ij, ij tries to shutdown the connection. The code relevant to this is found here org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416) here we do the following if (d!=null) { // do we have a driver running? shutdown on exit. try { DriverManager.getConnection("jdbc:derby:;shutdown=true"); } catch (SQLException e) { // ignore the errors, they are expected. } } now with authentication enabled trying to take a connection to shutdown like done in the above code will result in an exception ij> connect 'jdbc:derby:;shutdown=true'; ERROR 08004: Connection authentication failure occurred. Reason: Invalid authentication.. This is the reason why upon a quit; in ij with authentication enabled and in the embedded mode the modules are never shutdown.
        Hide
        V.Narayanan added a comment -

        We get a better picture of the above error if we replace the catch clause in

        org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416)

        if (d!=null) { // do we have a driver running? shutdown on exit.
        try

        { DriverManager.getConnection("jdbc:derby:;shutdown=true"); }

        catch (SQLException e)

        { // ignore the errors, they are expected. }

        }

        with a e.printStackTrace()

        I get the following everytime I try to take a connection and quit ij with
        authentication enabled

        ij version 10.5
        ij> connect 'jdbc:derby:mydb;user=oystein;password=pass';
        ij> quit;
        java.sql.SQLNonTransientConnectionException: Connection authentication failure occurred. Reason: Invalid authentication..
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:76)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:87)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:93)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:172)
        at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:220)
        at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:207)
        at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416)
        at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:249)
        at org.apache.derby.impl.tools.ij.Main.go(Main.java:215)
        at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:181)
        at org.apache.derby.impl.tools.ij.Main.main(Main.java:73)
        at org.apache.derby.tools.ij.main(ij.java:59)
        Caused by: java.sql.SQLException: Connection authentication failure occurred. Reason: Invalid authentication..
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:119)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:70)
        ... 13 more
        vn@vn-laptop:~/work/workspaces/PrintStackTrace$

        Show
        V.Narayanan added a comment - We get a better picture of the above error if we replace the catch clause in org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416) if (d!=null) { // do we have a driver running? shutdown on exit. try { DriverManager.getConnection("jdbc:derby:;shutdown=true"); } catch (SQLException e) { // ignore the errors, they are expected. } } with a e.printStackTrace() I get the following everytime I try to take a connection and quit ij with authentication enabled ij version 10.5 ij> connect 'jdbc:derby:mydb;user=oystein;password=pass'; ij> quit; java.sql.SQLNonTransientConnectionException: Connection authentication failure occurred. Reason: Invalid authentication.. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:76) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:87) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:93) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:172) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:220) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:207) at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:416) at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:249) at org.apache.derby.impl.tools.ij.Main.go(Main.java:215) at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:181) at org.apache.derby.impl.tools.ij.Main.main(Main.java:73) at org.apache.derby.tools.ij.main(ij.java:59) Caused by: java.sql.SQLException: Connection authentication failure occurred. Reason: Invalid authentication.. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:119) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:70) ... 13 more vn@vn-laptop:~/work/workspaces/PrintStackTrace$
        Hide
        V.Narayanan added a comment -

        I get an authentication exception similar to the one above in a client/server
        environment too (with the same modification to the derby codebase as above).

        I however don't think I can fault derby here.

        I believe when we call quit; on ij (after booting a database in the embedded mode)
        we are trying to shutdown ij without worrying about the database that we might
        have booted earlier. I do not think we can restrict the user from exiting (shutting
        down) the application (ij) he has started just because he has taken an authenticated
        derby connection inside the application(ij).

        This seems like a perfectly legal operation to me.

        ij relying on taking a successful system shutdown derby connection (without proper
        user name and password) when authentication is enabled causes the modules in the
        derby system to not be shutdown in a clean fashion.

        Basically ij tries its best to do a clean shutdown of the system upon exit.

        From the analysis above I am tempted to close this JIRA issue as invalid (unless someone
        has an objection).

        I however think this behavior of ij when we call a quit (of not doing a clean shutdown when authentication
        is enabled) can be documented somewhere (if not already documented). If the community feels the same
        way I will raise a JIRA requesting this documentation to be included.

        Show
        V.Narayanan added a comment - I get an authentication exception similar to the one above in a client/server environment too (with the same modification to the derby codebase as above). I however don't think I can fault derby here. I believe when we call quit; on ij (after booting a database in the embedded mode) we are trying to shutdown ij without worrying about the database that we might have booted earlier. I do not think we can restrict the user from exiting (shutting down) the application (ij) he has started just because he has taken an authenticated derby connection inside the application(ij). This seems like a perfectly legal operation to me. ij relying on taking a successful system shutdown derby connection (without proper user name and password) when authentication is enabled causes the modules in the derby system to not be shutdown in a clean fashion. Basically ij tries its best to do a clean shutdown of the system upon exit. From the analysis above I am tempted to close this JIRA issue as invalid (unless someone has an objection). I however think this behavior of ij when we call a quit (of not doing a clean shutdown when authentication is enabled) can be documented somewhere (if not already documented). If the community feels the same way I will raise a JIRA requesting this documentation to be included.
        Hide
        Jørgen Løland added a comment -

        I guess this (quiting derby without shutting down the databases) has other implications as well. E.g., when a booted database is not properly shut down, the checkpoint that is normally made on shutdown will not be there, increasing the time to recover the next time that database is booted.

        Combining what you say here with the discussion on 3447, my suggestion is twofold:

        • I agree that derby should not hang if the user shuts down an application without shutting down the databases. Your suggestion of using daemon threads for replication sounds good to me.
        • On the other hand, we control the ij application, and we may want to improve it. Documenting this problematic behavior like suggested may be enough, but it would also be possible to abort the 'quit;' call with an exception saying:

        ij > quit;
        Error: could not quit ij because the following databases have not been shut down: "test1", "test2". Shut down these databases first and then quit ij.

        I don't feel strongly about either option (only document / fail 'quit;'), but I think that in any case, the severity can be set lower now that we know the cause of the problem.

        Show
        Jørgen Løland added a comment - I guess this (quiting derby without shutting down the databases) has other implications as well. E.g., when a booted database is not properly shut down, the checkpoint that is normally made on shutdown will not be there, increasing the time to recover the next time that database is booted. Combining what you say here with the discussion on 3447, my suggestion is twofold: I agree that derby should not hang if the user shuts down an application without shutting down the databases. Your suggestion of using daemon threads for replication sounds good to me. On the other hand, we control the ij application, and we may want to improve it. Documenting this problematic behavior like suggested may be enough, but it would also be possible to abort the 'quit;' call with an exception saying: ij > quit; Error: could not quit ij because the following databases have not been shut down: "test1", "test2". Shut down these databases first and then quit ij. I don't feel strongly about either option (only document / fail 'quit;'), but I think that in any case, the severity can be set lower now that we know the cause of the problem.
        Hide
        V.Narayanan added a comment -

        >* On the other hand, we control the ij application, and we may want to improve it.
        >Documenting this problematic behavior like suggested may be enough, but it would
        >also be possible to abort the 'quit;' call with an exception saying:

        Depends on how much of derby specific code you want to put in ij, Jorgen.

        I must admit I was surprised at seeing the derby system shutdown being attempted
        when we do a quit itself.

        My impression about ij was ij is a general jdbc front end tool. It can be used with any database
        that supports a jdbc driver. This shutdown attempt with use to any other database (different
        jdbc url) will surely fail. Well maybe that is why we ignore the SQLExceptions upon doing
        this.

        Adding more and more derby specific code to ij, makes ij more and more a
        derby specific tool and lesser and lesser a general jdbc interface supporting front-end.

        For now I will lower the severity of the issue. I think we can take it from there.

        I will also leave the issue open.

        Show
        V.Narayanan added a comment - >* On the other hand, we control the ij application, and we may want to improve it. >Documenting this problematic behavior like suggested may be enough, but it would >also be possible to abort the 'quit;' call with an exception saying: Depends on how much of derby specific code you want to put in ij, Jorgen. I must admit I was surprised at seeing the derby system shutdown being attempted when we do a quit itself. My impression about ij was ij is a general jdbc front end tool. It can be used with any database that supports a jdbc driver. This shutdown attempt with use to any other database (different jdbc url) will surely fail. Well maybe that is why we ignore the SQLExceptions upon doing this. Adding more and more derby specific code to ij, makes ij more and more a derby specific tool and lesser and lesser a general jdbc interface supporting front-end. For now I will lower the severity of the issue. I think we can take it from there. I will also leave the issue open.
        Hide
        Øystein Grøvlen added a comment -

        I think it is unfortunate that ij behaves differently with and without authentication.

        I also feel that is very convenient that ij shut down the database when I exit. Lately, when I have done some work in ij on a database with authentication/authorization turned on, it has been very annoying that if I quit I ij after doing a large update operation, it has taken a long time to boot the database the next time. (Now I understand why)

        I see the following possible solutions:

        1. Change ij so the database is never shut down on quit, not just when authentication is turned on. Document that users need to shutdown databases before they exit ij, similarly to the message to JDBC programmers.
        2. Somehow fix the case with authentication so that the database is shut down. (I am not sure how)
        3. Let quit report error if ij is not able to shut down a database due to authentication. Introduce a new command that can be use to force quit ij without shutting down databases.

        Show
        Øystein Grøvlen added a comment - I think it is unfortunate that ij behaves differently with and without authentication. I also feel that is very convenient that ij shut down the database when I exit. Lately, when I have done some work in ij on a database with authentication/authorization turned on, it has been very annoying that if I quit I ij after doing a large update operation, it has taken a long time to boot the database the next time. (Now I understand why) I see the following possible solutions: 1. Change ij so the database is never shut down on quit, not just when authentication is turned on. Document that users need to shutdown databases before they exit ij, similarly to the message to JDBC programmers. 2. Somehow fix the case with authentication so that the database is shut down. (I am not sure how) 3. Let quit report error if ij is not able to shut down a database due to authentication. Introduce a new command that can be use to force quit ij without shutting down databases.
        Hide
        Tiago R. Espinha added a comment -

        Triaged for 10.5.2. No changes made.

        Show
        Tiago R. Espinha added a comment - Triaged for 10.5.2. No changes made.
        Hide
        Lily Wei added a comment -

        I tried this bug on trunk version(10.6). I can not reproduce this bug.
        $ ij -p derby.properties
        ij version 10.6
        ij> connect 'jdbc:derby:test;create=true;user=johndoe;password=pass';
        ij> quit;
        I am welcome to any suggestion to reproduce this bug.

        Show
        Lily Wei added a comment - I tried this bug on trunk version(10.6). I can not reproduce this bug. $ ij -p derby.properties ij version 10.6 ij> connect 'jdbc:derby:test;create=true;user=johndoe;password=pass'; ij> quit; I am welcome to any suggestion to reproduce this bug.
        Hide
        V.Narayanan added a comment -

        Thank you for trying out the reproducible.
        when you said "I can not reproduce this bug."
        did you mean that the modules are currently being
        closed? Are you sure that TopService#shutdown()
        is now being called with authentication enabled?

        Show
        V.Narayanan added a comment - Thank you for trying out the reproducible. when you said "I can not reproduce this bug." did you mean that the modules are currently being closed? Are you sure that TopService#shutdown() is now being called with authentication enabled?
        Hide
        Lily Wei added a comment -

        Thank you, too. This is a good bug to look at. I mean that when authentication is enable, quit does not do shutdown. However, I don't see how quit can shutdown when authentication is enable. In such case, shutdown process will hit authentication failure error first before successfully shutting down the database. I think that is the point 2 on Oystin's point if I may quote him.

        Show
        Lily Wei added a comment - Thank you, too. This is a good bug to look at. I mean that when authentication is enable, quit does not do shutdown. However, I don't see how quit can shutdown when authentication is enable. In such case, shutdown process will hit authentication failure error first before successfully shutting down the database. I think that is the point 2 on Oystin's point if I may quote him.
        Hide
        V.Narayanan added a comment -

        Oystein says:

        "2. Somehow fix the case with authentication so that the database is shut down. (I am not sure how)"

        Although he does mean that it is difficult to shutdown the database when authentication is enabled
        (the same way shutdown is done when authentication is not enabled) he does not mean this is not
        a bug.

        Like in my comments before, ij is a generic jdbc front end tool, I am not happy with a derby specific
        call here at all. Ideally speaking you should be capable of using ij with any DB that supports a jdbc
        driver.

        If everyone agrees we can probably document that we need to shutdown DB's before quitting and
        prevent ij from going down without closing the DB's. That would be one probable solution. But again
        I am not sure I will be happy with such a crude fix.

        Here is my 2 paisa

        (IJ is a simple and elegant tool. There is a innocent simplicity to the tool that I would never want it
        to lose. If a user wants a sophisticated front end there are other options. We should not clutter ij
        unnecessarily.)

        Show
        V.Narayanan added a comment - Oystein says: "2. Somehow fix the case with authentication so that the database is shut down. (I am not sure how)" Although he does mean that it is difficult to shutdown the database when authentication is enabled (the same way shutdown is done when authentication is not enabled) he does not mean this is not a bug. Like in my comments before, ij is a generic jdbc front end tool, I am not happy with a derby specific call here at all. Ideally speaking you should be capable of using ij with any DB that supports a jdbc driver. If everyone agrees we can probably document that we need to shutdown DB's before quitting and prevent ij from going down without closing the DB's. That would be one probable solution. But again I am not sure I will be happy with such a crude fix. Here is my 2 paisa (IJ is a simple and elegant tool. There is a innocent simplicity to the tool that I would never want it to lose. If a user wants a sophisticated front end there are other options. We should not clutter ij unnecessarily.)
        Hide
        Kathey Marsden added a comment -

        V. Narayanan said:
        >If everyone agrees we can probably document that we need to shutdown DB's before quitting and prevent ij from going down without closing the DB's.

        I think that is just too dangerous. There are a lot of applications that use ij scripts during their install that might hang if we do this. We do document that exit (quit) does a shutdown. http://db.apache.org/derby/docs/dev/tools/rtoolsijcomref33358.html

        I think the best we can do is document that with authentication, shutdown should be performed explicitly because the ij request will not succeed. I think even adding a warning that shutdown was not successful might cause automated scripts to fail.

        Show
        Kathey Marsden added a comment - V. Narayanan said: >If everyone agrees we can probably document that we need to shutdown DB's before quitting and prevent ij from going down without closing the DB's. I think that is just too dangerous. There are a lot of applications that use ij scripts during their install that might hang if we do this. We do document that exit (quit) does a shutdown. http://db.apache.org/derby/docs/dev/tools/rtoolsijcomref33358.html I think the best we can do is document that with authentication, shutdown should be performed explicitly because the ij request will not succeed. I think even adding a warning that shutdown was not successful might cause automated scripts to fail.

          People

          • Assignee:
            Unassigned
            Reporter:
            V.Narayanan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development