Derby
  1. Derby
  2. DERBY-4913

10.3 to 10.5 upgrade fails with ava.io.StreamCorruptedException: java.lang.ClassCastException: org.apache.derby.catalog.types.OldRoutineType incompatible with org.apache.derby.iapi.types.DataTypeDescriptor

    Details

    • Issue & fix info:
      Patch Available, Repro attached, Workaround attached
    • Bug behavior facts:
      Crash

      Description

      I have a report from a user upgrading to 10.5 from 10.3 that they got the following error during upgrade. I don't have much in the way of details yet, but thought I would post an issue since I've never seen this error before. I do have the original 10.3 database and it seems to upgrade fine to 10.5 with ij.

      java.sql.SQLException: Failed to start database
      '/
      org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException
      (Unknown Source)
      at
      org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown
      Source)
      at
      org.apache.derby.impl.jdbc.Util.seeNextException(Unknown
      Source)
      at
      org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown
      Source)
      at
      org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown
      Source)
      at
      org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown
      Source)
      at
      org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown
      Source)
      at
      org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown
      Source)
      at org.apache.derby.jdbc.InternalDriver.connect(Unknown
      Source)
      at
      org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown
      Source)
      at
      org.apache.derby.jdbc.EmbedPooledConnection.openRealConnection(U
      nknown Source)
      at
      org.apache.derby.jdbc.EmbedPooledConnection.<init>(Unknown
      Source)
      at
      org.apache.derby.jdbc.EmbedPooledConnection40.<init>(Unknown
      Source)
      at
      org.apache.derby.jdbc.Driver40.getNewPooledConnection(Unknown
      Source)
      at
      org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.createPoo
      ledConnection(Unknown Source)
      at
      org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.getPooled
      Connection(Unknown Source)
      at
      <snip>
      Caused by: java.sql.SQLException: Failed to start database
      '<snip>759243AF2F8
      4F1DE' with class loader <snip>.ExtClassLoader@3e955f6, see the next
      exception for details.
      at
      org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(U
      nknown Source)
      at
      org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTran
      sportAcrossDRDA(Unknown Source)
      ... 41 more
      Caused by: java.sql.SQLException: Exception during restore
      of a serializable or SQLData object of class
      at
      org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(U
      nknown Source)
      at
      org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTran
      sportAcrossDRDA(Unknown Source)
      at
      org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException
      (Unknown Source)
      at
      org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
      Source)
      ... 38 more
      Caused by: ERROR XSDA8: Exception during restore of a
      serializable or SQLData object of class
      at
      org.apache.derby.iapi.error.StandardException.newException(Unkno
      wn Source)
      at
      org.apache.derby.impl.store.raw.data.StoredPage.readRecordFromAr
      ray(Unknown Source)
      at
      org.apache.derby.impl.store.raw.data.StoredPage.restoreRecordFro
      mSlot(Unknown Source)
      at
      org.apache.derby.impl.store.raw.data.BasePage.fetchFromSlot(Unkn
      own Source)
      at
      org.apache.derby.impl.store.access.conglomerate.GenericScanContr
      oller.fetchRows(Unknown Source)
      at
      org.apache.derby.impl.store.access.heap.HeapScan.fetchNext(Unkno
      wn Source)
      at
      org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescript
      orViaHeap(Unknown Source)
      at
      org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getAllSPSDe
      scriptors(Unknown Source)
      at
      org.apache.derby.impl.sql.catalog.DataDictionaryImpl.dropJDBCMet
      adataSPSes(Unknown Source)
      at
      org.apache.derby.impl.sql.catalog.DataDictionaryImpl.updateMetad
      ataSPSes(Unknown Source)
      at
      org.apache.derby.impl.sql.catalog.DD_Version.doFullUpgrade(Unkno
      wn Source)
      at
      org.apache.derby.impl.sql.catalog.DD_Version.upgradeIfNeeded(Unk
      nown Source)
      at
      org.apache.derby.impl.sql.catalog.DataDictionaryImpl.loadDiction
      aryTables(Unknown Source)
      at
      org.apache.derby.impl.sql.catalog.DataDictionaryImpl.boot(Unknow
      n Source)
      at
      org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown
      Source)
      at
      org.apache.derby.impl.services.monitor.TopService.bootModule(Unk
      nown Source)
      at
      org.apache.derby.impl.services.monitor.BaseMonitor.startModule(U
      nknown Source)
      at
      org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule
      (Unknown Source)
      at org.apache.derby.impl.db.BasicDatabase.boot(Unknown
      Source)
      at
      org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown
      Source)
      at
      org.apache.derby.impl.services.monitor.TopService.bootModule(Unk
      nown Source)
      at
      org.apache.derby.impl.services.monitor.BaseMonitor.bootService(U
      nknown Source)
      at
      org.apache.derby.impl.services.monitor.BaseMonitor.startProvider
      Service(Unknown Source)
      at
      org.apache.derby.impl.services.monitor.BaseMonitor.findProviderA
      ndStartService(Unknown Source)
      at
      org.apache.derby.impl.services.monitor.BaseMonitor.startPersiste
      ntService(Unknown Source)
      at
      org.apache.derby.iapi.services.monitor.Monitor.startPersistentSe
      rvice(Unknown Source)
      ... 38 more
      Caused by: java.io.StreamCorruptedException:
      java.lang.ClassCastException:
      org.apache.derby.catalog.types.OldRoutineType incompatible with
      org.apache.derby.iapi.types.DataTypeDescriptor
      at
      org.apache.derby.iapi.services.io.FormatIdInputStream.readObject
      (Unknown Source)
      at
      org.apache.derby.impl.sql.GenericResultDescription.readExternal(
      Unknown Source)
      at
      org.apache.derby.iapi.services.io.FormatIdInputStream.readObject
      (Unknown Source)
      at
      org.apache.derby.impl.sql.GenericStorablePreparedStatement.readE
      xternal(Unknown Source)
      at
      org.apache.derby.iapi.services.io.FormatIdInputStream.readObject
      (Unknown Source)
      at
      org.apache.derby.iapi.types.UserType.readExternal(Unknown
      Source)
      ... 63 more

      1. 10_3work.sql
        0.6 kB
        Kathey Marsden
      2. ClassLoaderUpgrade.java
        2 kB
        Kathey Marsden
      3. derby.log
        32 kB
        Kathey Marsden
      4. derby-4913_diff.txt
        6 kB
        Kathey Marsden
      5. derby-4913_diff2.txt
        7 kB
        Kathey Marsden
      6. derby-4913_initcause_diff.txt
        0.7 kB
        Kathey Marsden
      7. derby-4913_testfix_diff.txt
        3 kB
        Kathey Marsden
      8. TestGetFunctions.java
        1 kB
        Kathey Marsden

        Issue Links

          Activity

          Hide
          Kathey Marsden added a comment - - edited

          A few more notes. Attempting the upgrade with an isolated URLClassLoader did not pop the issue, see attached ClassLoaderUpgrade.java for that attempt, nor did it reproduce for someone trying to use the same environment/software here in the US. (User seeing the problem is in Europe.) The Derby version being used is 10.5.3.1 - (957402).

          I have a copy of the the database after the problem occurs and it seems to be in a half upgraded state. The database won't boot with 10.3 with the message
          ERROR XSLAN: Database at C:<snip dbname> has an incompatib
          le format with the current version of the software. The database was created by or upgraded by version 10.5,

          If I connect with 10.5 does not have the new update statistics procedure in SYS.SYSALIASES.

          Happily if I reconnect with 10.5 with upgrade=true, the update statistics procedure is created.

          I don't actually see where the cast might occur in FormatIdInputStream.readObject , but do see this code and comment:

          catch (ClassCastException cce)

          { // We catch this here as it is usuall a user error. // they have readExternal (or SQLData) that doesn't match // the writeExternal. and thus the object read is of // the incorrect type, e.g. Integer i = (Integer) in.readObject(); throw new StreamCorruptedException(cce.toString()); }

          Perhaps the cce.toString() does not give the full nested exception.

          Show
          Kathey Marsden added a comment - - edited A few more notes. Attempting the upgrade with an isolated URLClassLoader did not pop the issue, see attached ClassLoaderUpgrade.java for that attempt, nor did it reproduce for someone trying to use the same environment/software here in the US. (User seeing the problem is in Europe.) The Derby version being used is 10.5.3.1 - (957402). I have a copy of the the database after the problem occurs and it seems to be in a half upgraded state. The database won't boot with 10.3 with the message ERROR XSLAN: Database at C:<snip dbname> has an incompatib le format with the current version of the software. The database was created by or upgraded by version 10.5, If I connect with 10.5 does not have the new update statistics procedure in SYS.SYSALIASES. Happily if I reconnect with 10.5 with upgrade=true, the update statistics procedure is created. I don't actually see where the cast might occur in FormatIdInputStream.readObject , but do see this code and comment: catch (ClassCastException cce) { // We catch this here as it is usuall a user error. // they have readExternal (or SQLData) that doesn't match // the writeExternal. and thus the object read is of // the incorrect type, e.g. Integer i = (Integer) in.readObject(); throw new StreamCorruptedException(cce.toString()); } Perhaps the cce.toString() does not give the full nested exception.
          Hide
          Kathey Marsden added a comment -

          ClassLoaderUpgrade.java is a program I used to see if this problem would reproduce with upgrade with a URLClassLoader. It did not. dbname and DERBY_LIB_DIR need to be changed for user environment.

          Show
          Kathey Marsden added a comment - ClassLoaderUpgrade.java is a program I used to see if this problem would reproduce with upgrade with a URLClassLoader. It did not. dbname and DERBY_LIB_DIR need to be changed for user environment.
          Hide
          Kathey Marsden added a comment -

          Patch to use initCause on the new StreamCorruptionException to include the ClassCastException stack trace. I will run suites.All, but don't think it actually exercises this code, so a visual check would be good.

          Show
          Kathey Marsden added a comment - Patch to use initCause on the new StreamCorruptionException to include the ClassCastException stack trace. I will run suites.All, but don't think it actually exercises this code, so a visual check would be good.
          Hide
          Lily Wei added a comment -

          +1 for use initCause on the new StreamCorruptionException to include the ClassCastException stack trace after visual check and compile on my client space.

          Show
          Lily Wei added a comment - +1 for use initCause on the new StreamCorruptionException to include the ClassCastException stack trace after visual check and compile on my client space.
          Hide
          Rick Hillegas added a comment -

          The serialization of data types seems to have changed in both 10.4 and 10.5. It would be interesting to know if you see this problem when upgrading the database from 10.3 to 10.4. If that works, try upgrading from 10.4 to 10.5.

          Show
          Rick Hillegas added a comment - The serialization of data types seems to have changed in both 10.4 and 10.5. It would be interesting to know if you see this problem when upgrading the database from 10.3 to 10.4. If that works, try upgrading from 10.4 to 10.5.
          Hide
          Kathey Marsden added a comment - - edited

          I got the nested exception back after the initcause patch which is

          ... 63 more
          Caused by: java.lang.ClassCastException: org.apache.derby.catalog.types.OldRoutineType incompatible with org.apache.derby.iapi.types.DataTypeDescriptor
          at org.apache.derby.impl.sql.GenericColumnDescriptor.readExternal(GenericColumnDescriptor.java:223)
          at org.apache.derby.iapi.services.io.FormatIdInputStream.readObject(FormatIdInputStream.java:126)
          ... 69 more

          • Read this object from a stream of stored objects.
            *
          • @param in read this.
            *
          • @exception IOException thrown on error
          • @exception ClassNotFoundException thrown on error
            */
            public void readExternal(ObjectInput in)
            throws IOException, ClassNotFoundException { FormatableHashtable fh = (FormatableHashtable)in.readObject(); name = (String)fh.get("name"); tableName = (String)fh.get("tableName"); schemaName = (String)fh.get("schemaName"); columnPos = fh.getInt("columnPos"); type = (DataTypeDescriptor)fh.get("type"); isAutoincrement = fh.getBoolean("isAutoincrement"); updatableByCursor = fh.getBoolean("updatableByCursor"); }

          Mike said this to me, which looks like a good clue:

          This kind of code exists: /**

          • Old releases (10.3 and before) wrote out the runtime
          • DataTypeDescriptor for routine parameter and return types.
          • 10.4 onwards (DERBY-2775) always writes out the catalog
          • type TypeDescriptor. Here we see what object was read from
          • disk and if it was the old type, now mapped to OldRoutineType,
          • we extract the catalog type and use that.
            *
          • @param onDiskType The object read that represents the type.
          • @return A type descriptor.
            */
            private static TypeDescriptor getStoredType(Object onDiskType) { if (onDiskType instanceof OldRoutineType) return ((OldRoutineType) onDiskType).getCatalogType(); return (TypeDescriptor) onDiskType; }

            key might be to figure out if this code should also be in GenericResultDescription.

          Still working to get a local reproduction.

          Show
          Kathey Marsden added a comment - - edited I got the nested exception back after the initcause patch which is ... 63 more Caused by: java.lang.ClassCastException: org.apache.derby.catalog.types.OldRoutineType incompatible with org.apache.derby.iapi.types.DataTypeDescriptor at org.apache.derby.impl.sql.GenericColumnDescriptor.readExternal(GenericColumnDescriptor.java:223) at org.apache.derby.iapi.services.io.FormatIdInputStream.readObject(FormatIdInputStream.java:126) ... 69 more Read this object from a stream of stored objects. * @param in read this. * @exception IOException thrown on error @exception ClassNotFoundException thrown on error */ public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException { FormatableHashtable fh = (FormatableHashtable)in.readObject(); name = (String)fh.get("name"); tableName = (String)fh.get("tableName"); schemaName = (String)fh.get("schemaName"); columnPos = fh.getInt("columnPos"); type = (DataTypeDescriptor)fh.get("type"); isAutoincrement = fh.getBoolean("isAutoincrement"); updatableByCursor = fh.getBoolean("updatableByCursor"); } Mike said this to me, which looks like a good clue: This kind of code exists: /** Old releases (10.3 and before) wrote out the runtime DataTypeDescriptor for routine parameter and return types. 10.4 onwards ( DERBY-2775 ) always writes out the catalog type TypeDescriptor. Here we see what object was read from disk and if it was the old type, now mapped to OldRoutineType, we extract the catalog type and use that. * @param onDiskType The object read that represents the type. @return A type descriptor. */ private static TypeDescriptor getStoredType(Object onDiskType) { if (onDiskType instanceof OldRoutineType) return ((OldRoutineType) onDiskType).getCatalogType(); return (TypeDescriptor) onDiskType; } key might be to figure out if this code should also be in GenericResultDescription. Still working to get a local reproduction.
          Hide
          Kathey Marsden added a comment -

          OldRoutineType was introduced with DERBY-2775 so linking this issue to that one.

          Show
          Kathey Marsden added a comment - OldRoutineType was introduced with DERBY-2775 so linking this issue to that one.
          Hide
          Kathey Marsden added a comment -

          Attached is a reproduction for this issue.
          Run the script 10_3work.sql with 10.3 and then the program TestGetFunctions with 10.5

          The test case can probably be narrowed down some. But the problem has been so elusive that I just wanted to get the repro posted before mucking with it more.

          Show
          Kathey Marsden added a comment - Attached is a reproduction for this issue. Run the script 10_3work.sql with 10.3 and then the program TestGetFunctions with 10.5 The test case can probably be narrowed down some. But the problem has been so elusive that I just wanted to get the repro posted before mucking with it more.
          Hide
          Kathey Marsden added a comment -

          In the second part of the reproduction, only the connection is important in TestGetFunctions. I can reproduce with just ij by
          1) Run 10_3work.sql with 10.3
          3) Connect with connect 'jdbc:derby:wombat;create=true;upgrade=true'; with 10.5

          Oddly the create=true attribute is important as I tried several times without it and could not reproduce. I am not sure why it would make a difference as the database is not being created.

          Show
          Kathey Marsden added a comment - In the second part of the reproduction, only the connection is important in TestGetFunctions. I can reproduce with just ij by 1) Run 10_3work.sql with 10.3 3) Connect with connect 'jdbc:derby:wombat;create=true;upgrade=true'; with 10.5 Oddly the create=true attribute is important as I tried several times without it and could not reproduce. I am not sure why it would make a difference as the database is not being created.
          Hide
          Kathey Marsden added a comment -

          I noticed if I change the Upgrade tests to specify both upgrade=true and create=true on hard upgrade I hit eight instances of this error, e.g.
          Index: java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/Pha
          seChanger.java
          ===================================================================
          — java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/PhaseC
          hanger.java (revision 1053401)
          +++ java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/PhaseC
          hanger.java (working copy)
          @@ -99,7 +99,7 @@

          case UpgradeChange.PH_HARD_UPGRADE:
          JDBCDataSource.setBeanProperty(ds, "connectionAttributes",

          • "upgrade=true");
            + "upgrade=true;create=true");
            break;
            default:
            break;

          I don't really want to change the upgrade tests in this way permanantly, but will try to figure out how to have a sparate PH_HARD_UPGRADE_WITH_CREATE phase to test this issue.

          Show
          Kathey Marsden added a comment - I noticed if I change the Upgrade tests to specify both upgrade=true and create=true on hard upgrade I hit eight instances of this error, e.g. Index: java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/Pha seChanger.java =================================================================== — java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/PhaseC hanger.java (revision 1053401) +++ java/testing/org/apache/derbyTesting/functionTests/tests/upgradeTests/PhaseC hanger.java (working copy) @@ -99,7 +99,7 @@ case UpgradeChange.PH_HARD_UPGRADE: JDBCDataSource.setBeanProperty(ds, "connectionAttributes", "upgrade=true"); + "upgrade=true;create=true"); break; default: break; I don't really want to change the upgrade tests in this way permanantly, but will try to figure out how to have a sparate PH_HARD_UPGRADE_WITH_CREATE phase to test this issue.
          Hide
          Kathey Marsden added a comment -

          Attached is a patch for this issue, derby-4913_diff.txt. In GenericColumnDescriptor we needed to handle the case where an OldRoutineType was stored rather than a DataTypeDescriptor.

          A few notes:

          • I did not investigate why specifying create with upgrade triggers a different code path.
          • Rather than making OldRoutineType public I just assume it is a OldRoutineType if it is not already a DataTypeDescriptor, since these are the only two options.
          • For testing, I just added another iteration of the upgrade tests, just for 10.3.3.0 with both upgrade and create set.
          Show
          Kathey Marsden added a comment - Attached is a patch for this issue, derby-4913_diff.txt. In GenericColumnDescriptor we needed to handle the case where an OldRoutineType was stored rather than a DataTypeDescriptor. A few notes: I did not investigate why specifying create with upgrade triggers a different code path. Rather than making OldRoutineType public I just assume it is a OldRoutineType if it is not already a DataTypeDescriptor, since these are the only two options. For testing, I just added another iteration of the upgrade tests, just for 10.3.3.0 with both upgrade and create set.
          Hide
          Kathey Marsden added a comment -

          Here is an updated patch derby-4913_diff2.txt. After talking to Lily on IRC, I made the folllowing changes:

          1) Improved the comment for getStoredDataTypeDescriptor
          2) Changed the upgrade tests to just always run with create for 10.3.3.0 and removed the extra run. We should get enough coverage for 10.3 without the create by running 10.3.1.4. This way we avoid adding extra time to the tests.

          Thanks Lily for looking at this. will commit this afternoon and start back porting tomorrow.

          Show
          Kathey Marsden added a comment - Here is an updated patch derby-4913_diff2.txt. After talking to Lily on IRC, I made the folllowing changes: 1) Improved the comment for getStoredDataTypeDescriptor 2) Changed the upgrade tests to just always run with create for 10.3.3.0 and removed the extra run. We should get enough coverage for 10.3 without the create by running 10.3.1.4. This way we avoid adding extra time to the tests. Thanks Lily for looking at this. will commit this afternoon and start back porting tomorrow.
          Hide
          Knut Anders Hatlen added a comment -

          Would it be better to check the contents of the version object instead of checking the offset? I think we have a way to run the upgrade tests against a specific set of old versions, in which case 10.3.3.0 may be at a different offset.

          Show
          Knut Anders Hatlen added a comment - Would it be better to check the contents of the version object instead of checking the offset? I think we have a way to run the upgrade tests against a specific set of old versions, in which case 10.3.3.0 may be at a different offset.
          Hide
          Kathey Marsden added a comment -

          Thank you Knut for looking at the patch. Here is the test fix to check the actual version and not the offset. I'll go ahead and check it in.

          Show
          Kathey Marsden added a comment - Thank you Knut for looking at the patch. Here is the test fix to check the actual version and not the offset. I'll go ahead and check it in.
          Hide
          Kathey Marsden added a comment -

          Merged fix and test from trunk back to 10.5 and fix only to 10.4. Earlier branches are not affected.

          Show
          Kathey Marsden added a comment - Merged fix and test from trunk back to 10.5 and fix only to 10.4. Earlier branches are not affected.
          Hide
          Kathey Marsden added a comment -

          Sounds good, but of course deregister=false will be the common usage to keep the driver loaded. true is the default so I am guessing would be rarely used. Thank you Kim for picking up the documentation issue!

          Show
          Kathey Marsden added a comment - Sounds good, but of course deregister=false will be the common usage to keep the driver loaded. true is the default so I am guessing would be rarely used. Thank you Kim for picking up the documentation issue!
          Hide
          Kathey Marsden added a comment -

          I think implementing this will entail invoking something similar to ExceptionUtil.dumpThreads and JVMInfo javaDump() in the test code.
          Then in SpawnedProcess.complete before destroying the process we would have to somehow trigger the thread dump of the process that is to be destroyed. I am not quite sure how to force the dump to occur in the spawned process.

          Show
          Kathey Marsden added a comment - I think implementing this will entail invoking something similar to ExceptionUtil.dumpThreads and JVMInfo javaDump() in the test code. Then in SpawnedProcess.complete before destroying the process we would have to somehow trigger the thread dump of the process that is to be destroyed. I am not quite sure how to force the dump to occur in the spawned process.

            People

            • Assignee:
              Kathey Marsden
              Reporter:
              Kathey Marsden
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development