Derby
  1. Derby
  2. DERBY-5531

Assert failure when inserting NULL into indexed column with territory-based collation

    Details

    • Urgency:
      Normal
    • Issue & fix info:
      Repro attached

      Description

      ij> connect 'jdbc:derby:colldb;territory=en;collation=TERRITORY_BASED;create=true';
      ij> create table t(x varchar(10) unique);
      0 rows inserted/updated/deleted
      ij> insert into t values null;
      ERROR XJ001: Java exception: 'ASSERT FAILED type of inserted column[0] = org.apache.derby.iapi.types.SQLVarchartype of template column[0] = org.apache.derby.iapi.types.CollatorSQLVarchar: org.apache.derby.shared.common.sanity.AssertFailure'.
      java.sql.SQLException: Java exception: 'ASSERT FAILED type of inserted column[0] = org.apache.derby.iapi.types.SQLVarchartype of template column[0] = org.apache.derby.iapi.types.CollatorSQLVarchar: org.apache.derby.shared.common.sanity.AssertFailure'.
      at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98)
      at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142)
      at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299)
      at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:436)
      at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:353)
      at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2288)
      at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:82)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1334)
      at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:630)
      at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:559)
      at org.apache.derby.impl.tools.ij.ij.executeImmediate(ij.java:367)
      at org.apache.derby.impl.tools.ij.utilMain.doCatch(utilMain.java:527)
      at org.apache.derby.impl.tools.ij.utilMain.runScriptGuts(utilMain.java:369)
      at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:245)
      at org.apache.derby.impl.tools.ij.Main.go(Main.java:229)
      at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:184)
      at org.apache.derby.impl.tools.ij.Main.main(Main.java:75)
      at org.apache.derby.tools.ij.main(ij.java:59)
      at org.apache.derby.iapi.tools.run.main(run.java:53)
      Caused by: java.sql.SQLException: Java exception: 'ASSERT FAILED type of inserted column[0] = org.apache.derby.iapi.types.SQLVarchartype of template column[0] = org.apache.derby.iapi.types.CollatorSQLVarchar: org.apache.derby.shared.common.sanity.AssertFailure'.
      at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:42)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122)
      at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71)
      ... 18 more
      Caused by: org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED type of inserted column[0] = org.apache.derby.iapi.types.SQLVarchartype of template column[0] = org.apache.derby.iapi.types.CollatorSQLVarchar
      at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162)
      at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147)
      at org.apache.derby.impl.store.access.btree.OpenBTree.isIndexableRowConsistent(OpenBTree.java:515)
      at org.apache.derby.impl.store.access.btree.BTreeController.doIns(BTreeController.java:679)
      at org.apache.derby.impl.store.access.btree.BTreeController.insert(BTreeController.java:1374)
      at org.apache.derby.impl.store.access.btree.index.B2IController.insert(B2IController.java:210)
      at org.apache.derby.impl.sql.execute.IndexChanger.insertAndCheckDups(IndexChanger.java:440)
      at org.apache.derby.impl.sql.execute.IndexChanger.doInsert(IndexChanger.java:383)
      at org.apache.derby.impl.sql.execute.IndexChanger.insert(IndexChanger.java:590)
      at org.apache.derby.impl.sql.execute.IndexSetChanger.insert(IndexSetChanger.java:268)
      at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:453)
      at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:999)
      at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:519)
      at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:443)
      at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:324)
      at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1242)
      ... 11 more

      1. DERBY5531_proposal1_diff.txt
        4 kB
        Mamta A. Satoor
      2. DERBY5531_proposal2_diff.txt
        3 kB
        Mamta A. Satoor

        Activity

        Hide
        Mamta A. Satoor added a comment -

        This jira has been backported upto 10.5.

        If anyone is in need of further backporting, they can backport it upto 10.3 which is where territory based collation was introduced.

        Show
        Mamta A. Satoor added a comment - This jira has been backported upto 10.5. If anyone is in need of further backporting, they can backport it upto 10.3 which is where territory based collation was introduced.
        Hide
        Mamta A. Satoor added a comment - - edited

        changes went in as revision 1378206 in trunk
        Backport went as revision 1378787 in 10.9 codelines.
        Backport went as revision 1378788 in 10.8 codelines.
        Backport went as revision 1380946 in 10.7 codelines.
        Backport went as revision 1380950 in 10.6 codelines.
        Backport went as revision 1381271 in 10.5 codelines.

        Show
        Mamta A. Satoor added a comment - - edited changes went in as revision 1378206 in trunk Backport went as revision 1378787 in 10.9 codelines. Backport went as revision 1378788 in 10.8 codelines. Backport went as revision 1380946 in 10.7 codelines. Backport went as revision 1380950 in 10.6 codelines. Backport went as revision 1381271 in 10.5 codelines.
        Hide
        Dag H. Wanvik added a comment -

        +1 to approach 2)

        Show
        Dag H. Wanvik added a comment - +1 to approach 2)
        Hide
        Mamta A. Satoor added a comment -

        I went through the code and have following 2 proposals.
        a)ResultColumnList.java's method columnTypesAndLengthMatch returns false if it finds a mismatch between column types and lengths.Based on this return value, the calling method InsertNode.bind determines if it needs a Normalize ResultSet. If the column has null value as it can be in the following example, the method columnTypesAndLengthMatch returns true even though the column length is not really the size of the underlying table's column.
        insert into t values null;
        We can change columnTypesAndLengthMatch to return false(as shown below) if the insert value is null but it will add additional runtime processing because of the Normalize ResultSet(I am attaching DERBY5531_proposal1_diff.txt which shows the proposed changes for this proposal).
        if((otherResultColumn.getExpression() instanceof ConstantNode) &&
        ((ConstantNode) otherResultColumn.getExpression()).isNull())

        { ConstantNode otherResultColumnConstNode = (ConstantNode)otherResultColumn.getExpression(); if (otherResultColumnConstNode.isNull()) return false; }

        I don't know the history behind why we have chosen not to have a normalize resultset if the values being inserted are null.This existing behavior of not having a normalize resultset for null values has not resulted in any bugs and hence I would prefer us not going with this proposal of adding normalize resultset on top of null insert values. I would prefer the 2nd proposal which is as follows.
        2)As per this proposal, we should change OpenBTree.isIndexableRowConsistent to not throw an assert failure if dealing with null values. If the value is null, then we should not assert the column datatypes. As it is, this method is meaningful only in sane mode. As we have seen in this jira, insane jars do not give any error for the test case provided in the jira. I am attaching
        DERBY5531_proposal2_diff.txt which shows the proposed changes for this proposal.

        I have added test cases in CollationTest.

        I have run derbyall and junit suite with both the proposals and there were no issues with the tests.

        Show
        Mamta A. Satoor added a comment - I went through the code and have following 2 proposals. a)ResultColumnList.java's method columnTypesAndLengthMatch returns false if it finds a mismatch between column types and lengths.Based on this return value, the calling method InsertNode.bind determines if it needs a Normalize ResultSet. If the column has null value as it can be in the following example, the method columnTypesAndLengthMatch returns true even though the column length is not really the size of the underlying table's column. insert into t values null; We can change columnTypesAndLengthMatch to return false(as shown below) if the insert value is null but it will add additional runtime processing because of the Normalize ResultSet(I am attaching DERBY5531_proposal1_diff.txt which shows the proposed changes for this proposal). if((otherResultColumn.getExpression() instanceof ConstantNode) && ((ConstantNode) otherResultColumn.getExpression()).isNull()) { ConstantNode otherResultColumnConstNode = (ConstantNode)otherResultColumn.getExpression(); if (otherResultColumnConstNode.isNull()) return false; } I don't know the history behind why we have chosen not to have a normalize resultset if the values being inserted are null.This existing behavior of not having a normalize resultset for null values has not resulted in any bugs and hence I would prefer us not going with this proposal of adding normalize resultset on top of null insert values. I would prefer the 2nd proposal which is as follows. 2)As per this proposal, we should change OpenBTree.isIndexableRowConsistent to not throw an assert failure if dealing with null values. If the value is null, then we should not assert the column datatypes. As it is, this method is meaningful only in sane mode. As we have seen in this jira, insane jars do not give any error for the test case provided in the jira. I am attaching DERBY5531_proposal2_diff.txt which shows the proposed changes for this proposal. I have added test cases in CollationTest. I have run derbyall and junit suite with both the proposals and there were no issues with the tests.
        Hide
        Mamta A. Satoor added a comment -

        It looks like at bind time of insert node, we go through the individual columns in the insert statement to determine if there is a need for normalization or not. For some reason, for our insert staement in this jira, we conclude that we do not need normalization and hence at execution time, the data type conversion from SQLVarchar to CollatorSQLVarchar never happens and hence the assert failure in OpenBTree.java. Here is the relevant code in InsertNode.bindStatement()
        /* Insert a NormalizeResultSetNode above the source if the source

        • and target column types and lengths do not match.
          */
          if (! resultColumnList.columnTypesAndLengthsMatch(
          resultSet.getResultColumns())) { resultSet = (ResultSetNode) getNodeFactory().getNode( C_NodeTypes.NORMALIZE_RESULT_SET_NODE, resultSet, resultColumnList, null, Boolean.FALSE, getContextManager()); }
        Show
        Mamta A. Satoor added a comment - It looks like at bind time of insert node, we go through the individual columns in the insert statement to determine if there is a need for normalization or not. For some reason, for our insert staement in this jira, we conclude that we do not need normalization and hence at execution time, the data type conversion from SQLVarchar to CollatorSQLVarchar never happens and hence the assert failure in OpenBTree.java. Here is the relevant code in InsertNode.bindStatement() /* Insert a NormalizeResultSetNode above the source if the source and target column types and lengths do not match. */ if (! resultColumnList.columnTypesAndLengthsMatch( resultSet.getResultColumns())) { resultSet = (ResultSetNode) getNodeFactory().getNode( C_NodeTypes.NORMALIZE_RESULT_SET_NODE, resultSet, resultColumnList, null, Boolean.FALSE, getContextManager()); }
        Hide
        Mamta A. Satoor added a comment -

        I have been debugging this issue and found that, for some reason, for a non-null value, we have a NormalizeResultSet which does the conversion for char to territory based char equivalent at execution time. But, for null value, we do not have NormalizeResultSet and hence no such conversion of data type happens. The conversion probably is not necessary since we are dealing with nulls. I am debugging further to see where the decision is happening about the requirement of NormalizeResultSet.

        Show
        Mamta A. Satoor added a comment - I have been debugging this issue and found that, for some reason, for a non-null value, we have a NormalizeResultSet which does the conversion for char to territory based char equivalent at execution time. But, for null value, we do not have NormalizeResultSet and hence no such conversion of data type happens. The conversion probably is not necessary since we are dealing with nulls. I am debugging further to see where the decision is happening about the requirement of NormalizeResultSet.
        Hide
        Kristian Waagan added a comment -

        Just noting that running with an insane build dosen't result in any error, neither in the insert nor in a subsequent select.

        Show
        Kristian Waagan added a comment - Just noting that running with an insane build dosen't result in any error, neither in the insert nor in a subsequent select.
        Hide
        Mike Matrigali added a comment -

        insert of a string works, so null is the special case causing the bug. At least at first look it looks like the calling code to store is wrong and that the type of column containing the null should be CollatorSQLVarchar.

        Show
        Mike Matrigali added a comment - insert of a string works, so null is the special case causing the bug. At least at first look it looks like the calling code to store is wrong and that the type of column containing the null should be CollatorSQLVarchar.
        Hide
        Mike Matrigali added a comment -

        repro'd for me also. Here is full stack on my system from derby.log:

        Tue Dec 13 12:11:11 PST 2011:
        Booting Derby version The Apache Software Foundation - Apache Derby - 10.9.0.0 alpha - (1): instance a816c00e-0134-390c-db37-00000001711d
        on database directory C:\mikem_db\colldb with class loader sun.misc.Launcher$AppClassLoader@69326932
        Loaded from file:/C:/derby/s1/classes/^M
        java.vendor=IBM Corporation
        java.runtime.version=jvmwi3260sr9-20101124_69295
        java.fullversion=JRE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260sr9-20101124_69295 (JIT enabled, AOT enabled)
        J9VM - 20101124_069295
        JIT - r9_20101028_17488ifx2
        GC - 20101027_AA
        user.dir=C:\mikem^M
        derby.system.home=_db^M
        Database Class Loader started - derby.database.classpath=''^M
        Tue Dec 13 12:11:49 PST 2011 Thread[main,5,main] (XID = 202), (SESSIONID = 1), (DATABASE = colldb), (DRDAID = null), Cleanup action starting^M
        Tue Dec 13 12:11:49 PST 2011 Thread[main,5,main] (XID = 202), (SESSIONID = 1), (DATABASE = colldb), (DRDAID = null), Failed Statement is: insert into
        t values null^M
        org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED type of inserted column[0] = org.apache.derby.iapi.types.SQLVarchartype of template
        column[0] = org.apache.derby.iapi.types.CollatorSQLVarchar^M
        at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162)^M
        at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147)^M
        at org.apache.derby.impl.store.access.btree.OpenBTree.isIndexableRowConsistent(OpenBTree.java:515)^M
        at org.apache.derby.impl.store.access.btree.BTreeController.doIns(BTreeController.java:679)^M
        at org.apache.derby.impl.store.access.btree.BTreeController.insert(BTreeController.java:1374)^M
        at org.apache.derby.impl.store.access.btree.index.B2IController.insert(B2IController.java:210)^M
        at org.apache.derby.impl.sql.execute.IndexChanger.insertAndCheckDups(IndexChanger.java:440)^M
        at org.apache.derby.impl.sql.execute.IndexChanger.doInsert(IndexChanger.java:383)^M
        at org.apache.derby.impl.sql.execute.IndexChanger.insert(IndexChanger.java:590)^M
        at org.apache.derby.impl.sql.execute.IndexSetChanger.insert(IndexSetChanger.java:268)^M
        at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:453)^M
        at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:999)^M
        at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:519)^M
        at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:443)^M
        at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:324)^M
        at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1242)^M
        at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:630)^M
        at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:559)^M
        at org.apache.derby.impl.tools.ij.ij.executeImmediate(ij.java:367)^M
        at org.apache.derby.impl.tools.ij.utilMain.doCatch(utilMain.java:527)^M
        at org.apache.derby.impl.tools.ij.utilMain.runScriptGuts(utilMain.java:369)^M
        at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:245)^M
        at org.apache.derby.impl.tools.ij.Main.go(Main.java:229)^M
        at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:184)^M
        at org.apache.derby.impl.tools.ij.Main.main(Main.java:75)^M
        at org.apache.derby.tools.ij.main(ij.java:59)^M

        Show
        Mike Matrigali added a comment - repro'd for me also. Here is full stack on my system from derby.log: Tue Dec 13 12:11:11 PST 2011: Booting Derby version The Apache Software Foundation - Apache Derby - 10.9.0.0 alpha - (1): instance a816c00e-0134-390c-db37-00000001711d on database directory C:\mikem_db\colldb with class loader sun.misc.Launcher$AppClassLoader@69326932 Loaded from file:/C:/derby/s1/classes/ ^M java.vendor=IBM Corporation java.runtime.version=jvmwi3260sr9-20101124_69295 java.fullversion=JRE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260sr9-20101124_69295 (JIT enabled, AOT enabled) J9VM - 20101124_069295 JIT - r9_20101028_17488ifx2 GC - 20101027_AA user.dir=C:\mikem^M derby.system.home=_db^M Database Class Loader started - derby.database.classpath=''^M Tue Dec 13 12:11:49 PST 2011 Thread [main,5,main] (XID = 202), (SESSIONID = 1), (DATABASE = colldb), (DRDAID = null), Cleanup action starting^M Tue Dec 13 12:11:49 PST 2011 Thread [main,5,main] (XID = 202), (SESSIONID = 1), (DATABASE = colldb), (DRDAID = null), Failed Statement is: insert into t values null^M org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED type of inserted column [0] = org.apache.derby.iapi.types.SQLVarchartype of template column [0] = org.apache.derby.iapi.types.CollatorSQLVarchar^M at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162)^M at org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147)^M at org.apache.derby.impl.store.access.btree.OpenBTree.isIndexableRowConsistent(OpenBTree.java:515)^M at org.apache.derby.impl.store.access.btree.BTreeController.doIns(BTreeController.java:679)^M at org.apache.derby.impl.store.access.btree.BTreeController.insert(BTreeController.java:1374)^M at org.apache.derby.impl.store.access.btree.index.B2IController.insert(B2IController.java:210)^M at org.apache.derby.impl.sql.execute.IndexChanger.insertAndCheckDups(IndexChanger.java:440)^M at org.apache.derby.impl.sql.execute.IndexChanger.doInsert(IndexChanger.java:383)^M at org.apache.derby.impl.sql.execute.IndexChanger.insert(IndexChanger.java:590)^M at org.apache.derby.impl.sql.execute.IndexSetChanger.insert(IndexSetChanger.java:268)^M at org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(RowChangerImpl.java:453)^M at org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore(InsertResultSet.java:999)^M at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:519)^M at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:443)^M at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:324)^M at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1242)^M at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:630)^M at org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:559)^M at org.apache.derby.impl.tools.ij.ij.executeImmediate(ij.java:367)^M at org.apache.derby.impl.tools.ij.utilMain.doCatch(utilMain.java:527)^M at org.apache.derby.impl.tools.ij.utilMain.runScriptGuts(utilMain.java:369)^M at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:245)^M at org.apache.derby.impl.tools.ij.Main.go(Main.java:229)^M at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:184)^M at org.apache.derby.impl.tools.ij.Main.main(Main.java:75)^M at org.apache.derby.tools.ij.main(ij.java:59)^M

          People

          • Assignee:
            Mamta A. Satoor
            Reporter:
            Knut Anders Hatlen
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development