Uploaded image for project: 'Derby'
  1. Derby
  2. DERBY-6884

SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.11.1.1
    • Fix Version/s: 10.13.1.0
    • Component/s: SQL
    • Labels:
      None
    • Urgency:
      Normal
    • Issue & fix info:
      Newcomer, Repro attached
    • Bug behavior facts:
      Seen in production

      Description

      Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. This is because ImportReadData.initExternalLobFile() is parsing the offset as an Integer.

      The stack trace and a program to reproduce are below.

      java.lang.NumberFormatException: For input string: "2147483770"
      at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.8.0_45]
      at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45]
      at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45]
      at org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.load.Import.getBlob(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source) ~[derby-10.11.1.1.jar:na]
      at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source) ~[derby-10.11.1.1.jar:na]
      ... 36 common frames omitted

      ==================================
      package blob;

      import java.io.BufferedInputStream;
      import java.io.File;
      import java.io.FileInputStream;
      import java.io.IOException;
      import java.sql.*;

      public final class DerbyIssue {
      // derby url
      public static final String DBURL = "jdbc:derby:testdb;create=true";
      // any random binary file such as a large image or document
      public static final String BLOB_DATA_FILE = "...";
      public static final String EXPORT_TABLE_FILE = "table-data";
      public static final String EXPORT_BLOB_FILE = "blob-data";

      public static void main(String... args) throws Exception

      { final DerbyIssue test = new DerbyIssue(); test.run(); }

      public void run() throws Exception {
      Class.forName("org.apache.derby.jdbc.ClientDriver").getConstructor().newInstance();

      try(final Connection con = DriverManager.getConnection(DBURL)) {
      try (final Statement stmt = con.createStatement())

      { stmt.execute("CREATE TABLE TESTBLOB(id BIGINT, content BLOB)"); }

      System.out.printf("inserting test data%n");

      try (final PreparedStatement pstmt = con.prepareStatement("INSERT INTO TESTBLOB (id, content) VALUES (?, ?)")) {
      long id = 1;
      long byteCount = 0;
      final File content = new File(BLOB_DATA_FILE);
      while (byteCount < Integer.MAX_VALUE) {
      insertBlob(pstmt, id, content);
      id++;
      byteCount += content.length();
      if (id % 100 == 0)

      { System.out.printf("%d%n", byteCount); }

      }
      insertBlob(pstmt, id, content);
      byteCount += content.length();

      System.out.printf("%d bytes written to testblob table%n", byteCount);
      }

      final File exportFile = new File(EXPORT_TABLE_FILE);
      final File blobFile = new File(EXPORT_BLOB_FILE);
      try (final CallableStatement stmt = con.prepareCall(
      "CALL SYSCS_UTIL.SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE (null, ?, ?, null, null, null, ?)"))

      { stmt.setString(1, "TESTBLOB"); stmt.setString(2, exportFile.toString()); stmt.setString(3, blobFile.toString()); stmt.execute(); }

      System.out.printf("testblob table exported%n");

      try (final Statement stmt = con.createStatement())

      { stmt.execute("TRUNCATE TABLE TESTBLOB"); }

      System.out.printf("testblob table truncated%n");

      try (final CallableStatement stmt = con.prepareCall(
      "CALL SYSCS_UTIL.SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE (null, ?, ?, null, null, null, 0)"))

      { stmt.setString(1, "TESTBLOB"); stmt.setString(2, exportFile.toString()); stmt.execute(); }

      System.out.printf("testblob data imported%n");
      }
      }

      private void insertBlob(PreparedStatement pstmt, long id, File content) throws IOException, SQLException {
      try(BufferedInputStream contentStream = new BufferedInputStream(new FileInputStream(content)))

      { pstmt.setLong(1, id); pstmt.setBinaryStream(2, contentStream); pstmt.executeUpdate(); }

      }
      }

      1. DerbyIssue.java
        3 kB
        Bryan Pendleton
      2. firstTryAtTest.diff
        3 kB
        Bryan Pendleton
      3. JustChangeOffset.diff
        9 kB
        Bryan Pendleton
      4. testForLargeDataSuite.diff
        7 kB
        Bryan Pendleton
      5. trivial.diff
        2 kB
        Bryan Pendleton

        Activity

        Hide
        bryanpendleton Bryan Pendleton added a comment -

        The problem reproduces for me, just as described, using the
        current head of trunk on Windows, with JDK 1.8.0_77-b03

        I attached a re-formatted version of the repro program, which
        was easier for me to read and follow, as "DerbyIssue.java".

        I also removed the explicit load of the Derby ClientDriver which
        appears to be unnecessary with the repro program, as it uses
        the EmbeddedDriver and hence can run with just derby.jar.

        Also, to be clear: to run the repro program, you need to edit
        the program text to replace the three dots in the next line with
        the name of a valid file in your test directory.

        public static final String BLOB_DATA_FILE = "...";

        I used a 75 MB PDF file that I happened to have sitting around.

        The program cleverly loops, counting the size of the blobs
        that it has inserted, until it has more than 2 GB of them, so it
        doesn't really matter what file you use, but you have to pick a file.

        It would be nice to figure out a clever way to have a smaller repro,
        as this repro takes several minutes to run on my system, but
        for the purposes of demonstrating the bug the repro was great – thanks!

        Show
        bryanpendleton Bryan Pendleton added a comment - The problem reproduces for me, just as described, using the current head of trunk on Windows, with JDK 1.8.0_77-b03 I attached a re-formatted version of the repro program, which was easier for me to read and follow, as "DerbyIssue.java". I also removed the explicit load of the Derby ClientDriver which appears to be unnecessary with the repro program, as it uses the EmbeddedDriver and hence can run with just derby.jar. Also, to be clear: to run the repro program, you need to edit the program text to replace the three dots in the next line with the name of a valid file in your test directory. public static final String BLOB_DATA_FILE = "..."; I used a 75 MB PDF file that I happened to have sitting around. The program cleverly loops, counting the size of the blobs that it has inserted, until it has more than 2 GB of them, so it doesn't really matter what file you use, but you have to pick a file. It would be nice to figure out a clever way to have a smaller repro, as this repro takes several minutes to run on my system, but for the purposes of demonstrating the bug the repro was great – thanks!
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        I'm not really familiar with the code in this part of Derby, but I
        took the famous approach of trying to "do the simplest thing
        that could possibly work", and made the trivial change to the
        "lobOffset" and "lobLength" fields in the ImportReadData class.

        Attached 'trivial.diff' is the result.

        With this patch applied, the test program passes.

        I'm not willing to say this is the correct answer; in particular,
        I had to down-cast the offset and length fields to int in order
        to pass them to ImportLobFile.getString(), which expects int
        values for accessing clob data.

        But it did make the test program pass, so maybe it's a
        start toward a fix of some sort.

        I did no other testing at all.

        Show
        bryanpendleton Bryan Pendleton added a comment - I'm not really familiar with the code in this part of Derby, but I took the famous approach of trying to "do the simplest thing that could possibly work", and made the trivial change to the "lobOffset" and "lobLength" fields in the ImportReadData class. Attached 'trivial.diff' is the result. With this patch applied, the test program passes. I'm not willing to say this is the correct answer; in particular, I had to down-cast the offset and length fields to int in order to pass them to ImportLobFile.getString(), which expects int values for accessing clob data. But it did make the test program pass, so maybe it's a start toward a fix of some sort. I did no other testing at all.
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        With the patch applied, I ran the tools test suite, and there were no failures.

        This suggests that a simple path forward might be:
        1) Convert the repro into a new test case in ImportExportLobTest.java
        2) Commit the new test case, and the trivial diff

        I do worry that there may be other similar problems lurking though, so I
        intend to (at least) produce a variant of the test case which uses a set of CLOB
        columns rather than a set of BLOB columns, to check to see if there is a CLOB
        variant of this job lurking.

        Show
        bryanpendleton Bryan Pendleton added a comment - With the patch applied, I ran the tools test suite, and there were no failures. This suggests that a simple path forward might be: 1) Convert the repro into a new test case in ImportExportLobTest.java 2) Commit the new test case, and the trivial diff I do worry that there may be other similar problems lurking though, so I intend to (at least) produce a variant of the test case which uses a set of CLOB columns rather than a set of BLOB columns, to check to see if there is a CLOB variant of this job lurking.
        Hide
        knutanders Knut Anders Hatlen added a comment -

        I agree that the casts from long to int look a bit suspicious. Wouldn't they end up as negative values? The test case doesn't seem to exercise that part of the code, so it's difficult to verify that it works correctly. (I replaced the modified lines in getClobColumnFromExtFileAsString() with "throw new RuntimeException()", and the test case still passed.)

        Show
        knutanders Knut Anders Hatlen added a comment - I agree that the casts from long to int look a bit suspicious. Wouldn't they end up as negative values? The test case doesn't seem to exercise that part of the code, so it's difficult to verify that it works correctly. (I replaced the modified lines in getClobColumnFromExtFileAsString() with "throw new RuntimeException()", and the test case still passed.)
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        I finally got some time to try to develop a "clob" version of the repro
        program, and, as I think we all expected, it fails in a similar fashion:

        Caused by: java.lang.NumberFormatException: For input string: "2147487744"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:583)
        at java.lang.Integer.parseInt(Integer.java:615)
        at org.apache.derby.impl.load.ImportReadData.initExternalLobFile(ImportReadData.java:1040)
        at org.apache.derby.impl.load.ImportReadData.getClobColumnFromExtFileAsString(ImportReadData.java:953)
        at org.apache.derby.impl.load.ImportAbstract.getString(ImportAbstract.java:167)
        at org.apache.derby.impl.load.Import.getString(Import.java:45)
        at org.apache.derby.iapi.types.SQLChar.setValueFromResultSet(SQLChar.java:1466)
        at org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(VTIResultSet.java:688)
        at org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(VTIResultSet.java:461)
        at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:287)
        at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(NormalizeResultSet.java:188)
        ... (I truncated the stack trace)

        So there is clearly more work to be done, to address the issues on the CLOB side.

        Show
        bryanpendleton Bryan Pendleton added a comment - I finally got some time to try to develop a "clob" version of the repro program, and, as I think we all expected, it fails in a similar fashion: Caused by: java.lang.NumberFormatException: For input string: "2147487744" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:583) at java.lang.Integer.parseInt(Integer.java:615) at org.apache.derby.impl.load.ImportReadData.initExternalLobFile(ImportReadData.java:1040) at org.apache.derby.impl.load.ImportReadData.getClobColumnFromExtFileAsString(ImportReadData.java:953) at org.apache.derby.impl.load.ImportAbstract.getString(ImportAbstract.java:167) at org.apache.derby.impl.load.Import.getString(Import.java:45) at org.apache.derby.iapi.types.SQLChar.setValueFromResultSet(SQLChar.java:1466) at org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(VTIResultSet.java:688) at org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(VTIResultSet.java:461) at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:287) at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(NormalizeResultSet.java:188) ... (I truncated the stack trace) So there is clearly more work to be done, to address the issues on the CLOB side.
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        Attached is my first try at writing a regression test for these problems.

        Unfortunately, although this regression test appears to demonstrate
        the problem with "clob" data when the external file exceeds Integer.MAX_VALUE
        in size, the test is problematic: it takes more than 1 hour to run.

        I hope that I can improve the test program, because obviously
        a test that takes an hour is not appropriate to put into our
        test suite.

        My first ideas are (a) to not commit so often, and (b) to write a
        smaller number of larger clob objects.

        I'll try some of those ideas, and see if the runtime of the test
        is improved at all.

        Once I get a reliable test, including the "blob" version should
        be straightforward.

        Show
        bryanpendleton Bryan Pendleton added a comment - Attached is my first try at writing a regression test for these problems. Unfortunately, although this regression test appears to demonstrate the problem with "clob" data when the external file exceeds Integer.MAX_VALUE in size, the test is problematic: it takes more than 1 hour to run. I hope that I can improve the test program, because obviously a test that takes an hour is not appropriate to put into our test suite. My first ideas are (a) to not commit so often, and (b) to write a smaller number of larger clob objects. I'll try some of those ideas, and see if the runtime of the test is improved at all. Once I get a reliable test, including the "blob" version should be straightforward.
        Hide
        mikem Mike Matrigali added a comment -

        there use to be a test suite for tests like this - especially clob/blob testing. Even if you make it run fast, not everyone/everywhere will have the disk space to run it. The suite was meant to at least be run once per release and more often by some nightly testing framework if possible - not sure if anyone runs it any more.

        Show
        mikem Mike Matrigali added a comment - there use to be a test suite for tests like this - especially clob/blob testing. Even if you make it run fast, not everyone/everywhere will have the disk space to run it. The suite was meant to at least be run once per release and more often by some nightly testing framework if possible - not sure if anyone runs it any more.
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        I think the test suite you might be referring to is the "largeDataTests"

        Unfortunately, all the references I can find to those tests are > 5 years old,
        and my memory of how to run the old "runall" tests is fading.

        Still, I agree with you in principle; I'll do more research here.

        Show
        bryanpendleton Bryan Pendleton added a comment - I think the test suite you might be referring to is the "largeDataTests" Unfortunately, all the references I can find to those tests are > 5 years old, and my memory of how to run the old "runall" tests is fading. Still, I agree with you in principle; I'll do more research here.
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        Maybe something like:

        java org.apache.derbyTesting.functionTests.harness.RunSuite largeData

        might work?

        Show
        bryanpendleton Bryan Pendleton added a comment - Maybe something like: java org.apache.derbyTesting.functionTests.harness.RunSuite largeData might work?
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        I believe I've got the test able to reproduce both the CLOB and BLOB issues.

        However, it still takes a very long time to run.

        And, I don't have enough disk space on my (virtual) machine to run the test anymore,
        so I'll need to add some more disk space.

        It's clear that these tests don't belong in the regular test suite, so I'll look into
        placing these tests into the "largeData" suite suggested by Mike.

        Show
        bryanpendleton Bryan Pendleton added a comment - I believe I've got the test able to reproduce both the CLOB and BLOB issues. However, it still takes a very long time to run. And, I don't have enough disk space on my (virtual) machine to run the test anymore, so I'll need to add some more disk space. It's clear that these tests don't belong in the regular test suite, so I'll look into placing these tests into the "largeData" suite suggested by Mike.
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        I've moved the test cases to their own test program, so that
        they won't be run by the regular test suites, but can still be
        run as desired.

        Just as Knut Anders predicted, with 'trivial.diff' applied, the
        behavior of the CLOB test case changes from the integer
        parse error to a "Negative seek offset" error, due to casting
        the value from a long to an int producing a negative value.

        I'll try turning my attention to that detail later; for now I just
        wanted to record the progress I'd made.

        A snip from the stack trace is below.

        Caused by: java.io.IOException: Negative seek offset
        at java.io.RandomAccessFile.seek(RandomAccessFile.java:555)
        at org.apache.derby.impl.load.ImportFileInputStream.seek(ImportLobFile.java:266)
        at org.apache.derby.impl.load.ImportLobFile.getString(ImportLobFile.java:132)
        at org.apache.derby.impl.load.ImportReadData.getClobColumnFromExtFileAsString(ImportReadData.java:959)

        Show
        bryanpendleton Bryan Pendleton added a comment - I've moved the test cases to their own test program, so that they won't be run by the regular test suites, but can still be run as desired. Just as Knut Anders predicted, with 'trivial.diff' applied, the behavior of the CLOB test case changes from the integer parse error to a "Negative seek offset" error, due to casting the value from a long to an int producing a negative value. I'll try turning my attention to that detail later; for now I just wanted to record the progress I'd made. A snip from the stack trace is below. Caused by: java.io.IOException: Negative seek offset at java.io.RandomAccessFile.seek(RandomAccessFile.java:555) at org.apache.derby.impl.load.ImportFileInputStream.seek(ImportLobFile.java:266) at org.apache.derby.impl.load.ImportLobFile.getString(ImportLobFile.java:132) at org.apache.derby.impl.load.ImportReadData.getClobColumnFromExtFileAsString(ImportReadData.java:959)
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        After thinking about it some more in the clear light of morning, I took
        a closer look at the reference pages for the CLOB and BLOB data types.

        Both data types are limited to 2GB as their maximum length.

        So I think the only actual problem here is the offset in the external
        file, which needs to be a long to allow for external files > 2GB in size.

        The 'JustChangeOffset.diff' patch contains a modification to the offset
        field only; the length field is left as an int.

        This makes the code diff very small.

        I'll run various tests and see how it behaves.

        Show
        bryanpendleton Bryan Pendleton added a comment - After thinking about it some more in the clear light of morning, I took a closer look at the reference pages for the CLOB and BLOB data types. Both data types are limited to 2GB as their maximum length. So I think the only actual problem here is the offset in the external file, which needs to be a long to allow for external files > 2GB in size. The 'JustChangeOffset.diff' patch contains a modification to the offset field only; the length field is left as an int. This makes the code diff very small. I'll run various tests and see how it behaves.
        Hide
        knutanders Knut Anders Hatlen added a comment -

        JustChangeOffset.diff looks good to me. Except that jardriftcheck fails when building the jar files, because of the new class in derbyTesting.jar. +1 to commit when that's fixed.

        A couple of nits:

        --- java/engine/org/apache/derby/impl/load/ImportLobFile.java	(revision 1741376)
        +++ java/engine/org/apache/derby/impl/load/ImportLobFile.java	(working copy)
        @@ -128,7 +128,7 @@
              * @param length  length of the the data.
              * @exception  IOException  on any I/O error.     
              */
        -    public String getString(int offset, int length) throws IOException {
        +    public String getString(long offset, int length) throws IOException {
                 lobInputStream.seek(offset);
                 lobLimitIn.clearLimit();
                 lobLimitIn.setLimit((int) length);
        

        While you're at it, maybe also remove the redundant cast of length to int in the above call to setLimit(), so that readers don't have to spend cycles figuring out what the purpose of the cast is?

        You might also want to clean up the indentation in the test case. It uses a mix of tabs and spaces, and it doesn't always seem to agree with itself if tabs are 4 or 8 characters wide.

        +        PreparedStatement ps = getConnection().prepareStatement(
        +                     "insert into DERBY_6884_TESTCLOB values(? , ?)" );
        

        BaseJDBCTestCase has a helper method for preparing statments, so the above could have been replaced with the slightly simpler

                PreparedStatement ps = prepareStatement(
                             "insert into DERBY_6884_TESTCLOB values(? , ?)" );
        

        Then you don't have to close the prepared statement manually in the test case.

        Show
        knutanders Knut Anders Hatlen added a comment - JustChangeOffset.diff looks good to me. Except that jardriftcheck fails when building the jar files, because of the new class in derbyTesting.jar. +1 to commit when that's fixed. A couple of nits: --- java/engine/org/apache/derby/impl/load/ImportLobFile.java (revision 1741376) +++ java/engine/org/apache/derby/impl/load/ImportLobFile.java (working copy) @@ -128,7 +128,7 @@ * @param length length of the the data. * @exception IOException on any I/O error. */ - public String getString(int offset, int length) throws IOException { + public String getString(long offset, int length) throws IOException { lobInputStream.seek(offset); lobLimitIn.clearLimit(); lobLimitIn.setLimit((int) length); While you're at it, maybe also remove the redundant cast of length to int in the above call to setLimit(), so that readers don't have to spend cycles figuring out what the purpose of the cast is? You might also want to clean up the indentation in the test case. It uses a mix of tabs and spaces, and it doesn't always seem to agree with itself if tabs are 4 or 8 characters wide. + PreparedStatement ps = getConnection().prepareStatement( + "insert into DERBY_6884_TESTCLOB values(? , ?)" ); BaseJDBCTestCase has a helper method for preparing statments, so the above could have been replaced with the slightly simpler PreparedStatement ps = prepareStatement( "insert into DERBY_6884_TESTCLOB values(? , ?)" ); Then you don't have to close the prepared statement manually in the test case.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1742057 from Bryan Pendleton in branch 'code/trunk'
        [ https://svn.apache.org/r1742057 ]

        DERBY-6884: SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import lob data

        This change modifies the ImportLobFile.getString() and
        ImportReadData.initExternalLobFile() methods so that they use a Java "long"
        variable for the offset into the external lob file; prior to this change
        they were using a Java "int" variable and hence would malfunction when the
        lob offsets exceeded Integer.MAX_VALUE ( 2,147,483,647 ).

        The regression test which demonstrates these problems is a bit slow to run;
        on my system, it takes approximately 15 minutes to execute, and requires
        about 10 GB of available disk space during the test run. Therefore, the
        test cases are placed in a new test program (Derby6884Test), which is not
        listed in the "standard" system test suites, but rather is only added to the
        "largedata" suite. The new test can also be run by itself, e.g.:

        ant -Dderby.junit.testclass=org.apache.derbyTesting.functionTests.tests.largedata.Derby6884Test junit-clean junit-single

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1742057 from Bryan Pendleton in branch 'code/trunk' [ https://svn.apache.org/r1742057 ] DERBY-6884 : SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import lob data This change modifies the ImportLobFile.getString() and ImportReadData.initExternalLobFile() methods so that they use a Java "long" variable for the offset into the external lob file; prior to this change they were using a Java "int" variable and hence would malfunction when the lob offsets exceeded Integer.MAX_VALUE ( 2,147,483,647 ). The regression test which demonstrates these problems is a bit slow to run; on my system, it takes approximately 15 minutes to execute, and requires about 10 GB of available disk space during the test run. Therefore, the test cases are placed in a new test program (Derby6884Test), which is not listed in the "standard" system test suites, but rather is only added to the "largedata" suite. The new test can also be run by itself, e.g.: ant -Dderby.junit.testclass=org.apache.derbyTesting.functionTests.tests.largedata.Derby6884Test junit-clean junit-single
        Hide
        bryanpendleton Bryan Pendleton added a comment -

        Thanks Knut Anders for the careful review and good suggestions.

        The blank/tab thing continues to confound my editor settings, I'm afraid.
        Since the test program is entirely new source, I decided to use only
        spaces in that (small) source file. That way, it is at least internally
        consistent.

        The code changes in question are small, and could easily be
        back-ported to earlier releases, but I am not planning to do that
        at this time.

        Show
        bryanpendleton Bryan Pendleton added a comment - Thanks Knut Anders for the careful review and good suggestions. The blank/tab thing continues to confound my editor settings, I'm afraid. Since the test program is entirely new source, I decided to use only spaces in that (small) source file. That way, it is at least internally consistent. The code changes in question are small, and could easily be back-ported to earlier releases, but I am not planning to do that at this time.

          People

          • Assignee:
            bryanpendleton Bryan Pendleton
            Reporter:
            ehowe Edward Howe
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development