Derby
  1. Derby
  2. DERBY-5530

SQLChar.getCollationKey NPE in index-stat-thread

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.7.1.1, 10.8.2.2, 10.9.1.0
    • Fix Version/s: 10.7.1.4, 10.8.3.0, 10.9.1.0
    • Component/s: SQL
    • Labels:
      None
    • Environment:
      Windows 7 - NetBeans 6.9.1
    • Urgency:
      Normal
    • Issue & fix info:
      Repro attached, Workaround attached
    • Bug behavior facts:
      Crash

      Description

      With this JDBC connection url is : jdbc:derby:directory:db_name;territory=fr_FR;collation=TERRITORY_BASED:PRIMARY;create=true
      I get a NullPointerException in index-stat-thread

      Sun Dec 11 19:33:11 CET 2011 Thread[pool-3-thread-1,5,main]

      {istat} "PROXIFLEX"."IDAXX_RES": update scheduled, reason=[no stats, row-estimate=375] (queueSize=1)
      Sun Dec 11 19:33:11 CET 2011 Thread[index-stat-thread,5,main] {istat,trace@26130360} worker thread started (xid=12049) [q/p/s=1/0/1,err:k/u/c=0/0/0,rej:f/d/o=0/0/0]
      Sun Dec 11 19:33:11 CET 2011 Thread[index-stat-thread,5,main] {istat,trace@26130360} processing "PROXIFLEX"."IDAXX_RES"
      Sun Dec 11 19:33:11 CET 2011 Thread[index-stat-thread,5,main] {istat}

      runtime exception during normal operation
      java.lang.NullPointerException
      at org.apache.derby.iapi.types.SQLChar.getCollationKey(Unknown Source)
      at org.apache.derby.iapi.types.WorkHorseForCollatorDatatypes.stringCompare(Unknown Source)
      at org.apache.derby.iapi.types.CollatorSQLVarchar.stringCompare(Unknown Source)
      at org.apache.derby.iapi.types.SQLChar.compare(Unknown Source)
      at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl$KeyComparator.compareWithPrevKey(Unknown Source)
      at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.updateIndexStatsMinion(Unknown Source)
      at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.generateStatistics(Unknown Source)
      at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.processingLoop(Unknown Source)
      at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.run(Unknown Source)
      at java.lang.Thread.run(Thread.java:662)
      Sun Dec 11 19:33:11 CET 2011 Thread[index-stat-thread,5,main]

      {istat,trace@26130360}

      worker thread exit [q/p/s=0/0/1,err:k/u/c=0/0/0,rej:f/d/o=0/0/0]

      If I remove territory and collation parameters I don't have the exception.

      In case you want to disable automatic statistics, set derby property : derby.storage.indexStats.auto to false.

      1. derby-5530-1a-propagate_collation_info.diff
        4 kB
        Kristian Waagan
      2. repro-debug.log
        6 kB
        Dag H. Wanvik
      3. repro.log
        9 kB
        Dag H. Wanvik
      4. repro.sh
        0.4 kB
        Dag H. Wanvik
      5. script.sh
        0.3 kB
        Dag H. Wanvik

        Activity

        Hide
        Kristian Waagan added a comment -

        I'm unable to reproduce this (modified the istat test to run with a collated database).
        Can you confirm that this is still a problem for you, and/or provide some more detail?
        For instance:
        o JVM vendor/version
        o relevant part of the database schema, i.e. column types and indexes/constraints
        o istat lines from derby.log with derby.storage.indexStats.log=true and derby.storage.indexStats.debug.trace=true (grep for "

        {istat}

        ")
        Note: If the lines you included in the report are the only ones, ignore this.

        I tried to reproduce on Solaris 11 with 1.6.0_26-b03 and 1.7.0_02-b13.

        Show
        Kristian Waagan added a comment - I'm unable to reproduce this (modified the istat test to run with a collated database). Can you confirm that this is still a problem for you, and/or provide some more detail? For instance: o JVM vendor/version o relevant part of the database schema, i.e. column types and indexes/constraints o istat lines from derby.log with derby.storage.indexStats.log=true and derby.storage.indexStats.debug.trace=true (grep for " {istat} ") Note: If the lines you included in the report are the only ones, ignore this. I tried to reproduce on Solaris 11 with 1.6.0_26-b03 and 1.7.0_02-b13.
        Hide
        Jean-Yves LINET added a comment - - edited

        Well I spent several more hours on this problem trying to isolate the bug context. It was not obvious must I finally found how to reproduce it.
        In fact the bug is quite easy to reproduce and it comes from the use of the TRUNCATE TABLE statement.
        1- Create a table with a single column,
        2- Add an index on this column,
        3- Execute TRUNCATE statement on this table,
        4- Add at least two rows,
        5- Execute the SYSCS_UPDATE_STATISTICS procedure.
        If you have created the database with collation you have the NullPointerException, with no collation it is working fine.

        One more information. If you call the SYSCS_COMPRESS_TABLE before the statistics you don't have the NPE with collation database.

        I hope this information will be helpful to fix the bug.
        If you need more information just ask.

        Forgot to say that the derby.storage.indexStats.auto is disabled (false).

        Show
        Jean-Yves LINET added a comment - - edited Well I spent several more hours on this problem trying to isolate the bug context. It was not obvious must I finally found how to reproduce it. In fact the bug is quite easy to reproduce and it comes from the use of the TRUNCATE TABLE statement. 1- Create a table with a single column, 2- Add an index on this column, 3- Execute TRUNCATE statement on this table, 4- Add at least two rows, 5- Execute the SYSCS_UPDATE_STATISTICS procedure. If you have created the database with collation you have the NullPointerException, with no collation it is working fine. One more information. If you call the SYSCS_COMPRESS_TABLE before the statistics you don't have the NPE with collation database. I hope this information will be helpful to fix the bug. If you need more information just ask. Forgot to say that the derby.storage.indexStats.auto is disabled (false).
        Hide
        Dag H. Wanvik added a comment -

        Hi Jean-Yves,
        I tried to reproduce this using the attached script. It did not give NPE for me. Have a look and see if you can modify it to show the error.

        Show
        Dag H. Wanvik added a comment - Hi Jean-Yves, I tried to reproduce this using the attached script. It did not give NPE for me. Have a look and see if you can modify it to show the error.
        Hide
        Jean-Yves LINET added a comment -

        Hi,
        Well you have to use a VARCHAR column as collation are used with string values, not int !

        Show
        Jean-Yves LINET added a comment - Hi, Well you have to use a VARCHAR column as collation are used with string values, not int !
        Hide
        Dag H. Wanvik added a comment -

        Thanks for your help, Jean-Yves, I now have a working repro, uploaded: repro.sh (creates repro.sql) and resulting repro.log.
        (run with 10.8).

        I also enclose a run with a debug version of trunk which shows ASSERTION errors in debug mode. (repro-debug)

        Show
        Dag H. Wanvik added a comment - Thanks for your help, Jean-Yves, I now have a working repro, uploaded: repro.sh (creates repro.sql) and resulting repro.log. (run with 10.8). I also enclose a run with a debug version of trunk which shows ASSERTION errors in debug mode. (repro-debug)
        Hide
        Dag H. Wanvik added a comment -

        Not that with the debug version from trunk, the error arises in the INSERT; and ASSERT error.
        In the 10.8 non-debug run, I see the NPE in the call to SYSCS_COMPRESS_TABLE as you see.

        Show
        Dag H. Wanvik added a comment - Not that with the debug version from trunk, the error arises in the INSERT; and ASSERT error. In the 10.8 non-debug run, I see the NPE in the call to SYSCS_COMPRESS_TABLE as you see.
        Hide
        Kristian Waagan added a comment -

        Some comments on this issue:
        o the bug is in the truncate table code
        o the istat deamon and the update statistics code are not causing this bug
        o workaround: drop and recreate the index(es) (preferably just after the truncate command has finished)

        Show
        Kristian Waagan added a comment - Some comments on this issue: o the bug is in the truncate table code o the istat deamon and the update statistics code are not causing this bug o workaround: drop and recreate the index(es) (preferably just after the truncate command has finished)
        Hide
        Kristian Waagan added a comment -

        Another thing: TRUNCATE TABLE wasn't enabled before 10.7. Was the code used internally for something before that? If not I have to update the affected versions, since even though the code with the bug went into 10.3, there would be no way to run it (in producation/insane builds) until 10.7.

        Show
        Kristian Waagan added a comment - Another thing: TRUNCATE TABLE wasn't enabled before 10.7. Was the code used internally for something before that? If not I have to update the affected versions, since even though the code with the bug went into 10.3, there would be no way to run it (in producation/insane builds) until 10.7.
        Hide
        Jean-Yves LINET added a comment -

        Dropping and recreating index need to have the DDL of the indexes when truncate is called. So the truncate call can't be generic.
        I found that calling SYSCS_COMPRESS_TABLE just after the truncate was also a workaround.
        What could be the side effects of this solution compare with dropping and recreating index.

        Show
        Jean-Yves LINET added a comment - Dropping and recreating index need to have the DDL of the indexes when truncate is called. So the truncate call can't be generic. I found that calling SYSCS_COMPRESS_TABLE just after the truncate was also a workaround. What could be the side effects of this solution compare with dropping and recreating index.
        Hide
        Kristian Waagan added a comment -

        SYSCS_COMPRESS_TABLE drops and recreates the table/indexes for you, so that's a perfectly valid workaround.
        Note that SYSCS_INPLACE_COMPRESS_TABLE probably won't work.

        I have a patch for this bug, I'm running tests on it now.

        Show
        Kristian Waagan added a comment - SYSCS_COMPRESS_TABLE drops and recreates the table/indexes for you, so that's a perfectly valid workaround. Note that SYSCS_INPLACE_COMPRESS_TABLE probably won't work. I have a patch for this bug, I'm running tests on it now.
        Hide
        Kristian Waagan added a comment -

        Attaching patch 1a, which fixes the bug in truncate table.

        The bug was simply that the collation information wasn't propagated to the indexes.
        I've added two tests to CollationTest2.

        suites.All passed without errors.
        Patch ready for review.

        Show
        Kristian Waagan added a comment - Attaching patch 1a, which fixes the bug in truncate table. The bug was simply that the collation information wasn't propagated to the indexes. I've added two tests to CollationTest2. suites.All passed without errors. Patch ready for review.
        Hide
        Knut Anders Hatlen added a comment -

        +1

        Looks like a simple fix that makes truncateTable() pass collation information the same way as the other parts of AlterTableConstantAction.

        Nit: You probably meant to make the two new methods in CollationTest2 private?

        Show
        Knut Anders Hatlen added a comment - +1 Looks like a simple fix that makes truncateTable() pass collation information the same way as the other parts of AlterTableConstantAction. Nit: You probably meant to make the two new methods in CollationTest2 private?
        Hide
        Kristian Waagan added a comment -

        Thanks to Jean-Yves and Dag for helping with the repro.
        Thanks to Knut Anders for reviewing the patch.

        Regarding public vs private - yes they should be private. I changed this before committing.
        The reason they were public was that I assumed the JUnit pattern was followed in CollationTest2, but it does things a little differently (it lumps many tests together into one bigger test method).

        Committed patch 1a to trunk with revision 1243878.
        I'll leave this issue open for backporting.

        Show
        Kristian Waagan added a comment - Thanks to Jean-Yves and Dag for helping with the repro. Thanks to Knut Anders for reviewing the patch. Regarding public vs private - yes they should be private. I changed this before committing. The reason they were public was that I assumed the JUnit pattern was followed in CollationTest2, but it does things a little differently (it lumps many tests together into one bigger test method). Committed patch 1a to trunk with revision 1243878. I'll leave this issue open for backporting.
        Hide
        Kristian Waagan added a comment -

        Merged fix to 10.8 with revision 1292681, and to the 10.7 branch with revision 1292682.
        Resolving as fixed.

        Show
        Kristian Waagan added a comment - Merged fix to 10.8 with revision 1292681, and to the 10.7 branch with revision 1292682. Resolving as fixed.
        Hide
        Knut Anders Hatlen added a comment -

        [bulk update: close all resolved issues that haven't had any activity the last year]

        Show
        Knut Anders Hatlen added a comment - [bulk update: close all resolved issues that haven't had any activity the last year]

          People

          • Assignee:
            Kristian Waagan
            Reporter:
            Jean-Yves LINET
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development