Derby
  1. Derby
  2. DERBY-4477

Selecting / projecting a column whose value is represented by a stream more than once fails

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.3.3.0, 10.4.2.0, 10.5.3.0
    • Fix Version/s: 10.5.3.1, 10.6.1.0
    • Component/s: SQL, Store
    • Labels:
      None
    • Bug behavior facts:
      Crash, Data corruption, Wrong query result

      Description

      Selecting / projecting a column whose value is represented as a stream more than once crashes Derby, i.e.:
      ResultSet rs = stmt.executeQuery("SELECT clobValue AS clobOne, clobValue AS clobTwo FROM mytable");
      rs.getString(1);
      rs.getString(2);

      After having looked at the class of bugs having to do with reuse of stream data types, I now have a possible fix. It fixes DERBY-3645, DERBY-3646 and DERBY-2349 (there may be more Jiras).
      The core of the fix is cloning certain DVDs being selected/projected in multiple columns. There are two types of cloning:
      A) materializing clone
      B) stream clone

      (A) can be implemented already, (B) requires code to clone a stream without materializing it. Note that the streams I'm talking about are streams originating from the store.

      Testing revealed the following:

      • the cost of the checks performed to figure out if cloning is required seems acceptable (negligible?)
      • in some cases (A) has better performance than (B) because the raw data only has to be decoded once
      • stream clones are preferred when the data value is above a certain size for several reasons:
      • avoids potential out-of-memory errors (and in case of a server environment, it lowers the memory pressure)
      • avoids decoding the whole value if the JDBC streaming APIs are used to access only parts of the value
      • avoids decoding overall in cases where the value isn't accessed by the client / user
        (this statement conflicts with the performance observation above)

      We don't always know the size of a value, and since the fix code deals with all kinds of data types, it is slightly more costly to try to obtain the size.

      What do people think about the following goal statement?
      Goals:
      ----- Phase 1
      1) No crashes or wrong results due to stream reuse when executing duplicate column selections (minus goal 4)
      2) Minimal performance degradation for non-duplicate column selections
      3) Only a minor performance degradation for duplicate [[LONG] VAR]CHAR [FOR BIT DATA] column selections
      ----- Phase 2
      4) No out-of-memory exceptions during execution of duplicate column selections of BLOB/CLOB
      5) Optimize BLOB/CLOB cloning

      I think phase 1 can proceed by reviewing and discussing the prototype patch. Phase 2 requires more discussion and work (see DERBY-3650).

      A note about the bug behavior facts:
      Since this issue is the underlying cause for several other reported issues, I have decided to be liberal when setting the bug behavior facts. Depending on where the duplicate column selection is used, it can cause both crashes, wrong results and data corruption.

      1. derby-4477-useCloning.stat
        0.3 kB
        Dag H. Wanvik
      2. derby-4477-useCloning.diff
        3 kB
        Dag H. Wanvik
      3. derby-4477-partial-2.stat
        0.5 kB
        Dag H. Wanvik
      4. derby-4477-partial-2.diff
        23 kB
        Dag H. Wanvik
      5. derby-4477-partial.stat
        0.5 kB
        Dag H. Wanvik
      6. derby-4477-partial.diff
        14 kB
        Dag H. Wanvik
      7. derby-4477-lowmem-followup.stat
        0.2 kB
        Dag H. Wanvik
      8. derby-4477-lowmem-followup.diff
        2 kB
        Dag H. Wanvik
      9. derby-4477-lowmem-2.stat
        0.4 kB
        Dag H. Wanvik
      10. derby-4477-lowmem-2.diff
        8 kB
        Dag H. Wanvik
      11. derby-4477-lowmem.stat
        0.2 kB
        Dag H. Wanvik
      12. derby-4477-lowmem.diff
        7 kB
        Dag H. Wanvik
      13. derby-4477-0a-prototype.diff
        26 kB
        Kristian Waagan

        Issue Links

          Activity

          Kristian Waagan created issue -
          Kristian Waagan made changes -
          Field Original Value New Value
          Attachment derby-4477-0a-prototype.diff [ 12428056 ]
          Dag H. Wanvik made changes -
          Attachment derby-4477-partial.diff [ 12431064 ]
          Attachment derby-4477-partial.stat [ 12431065 ]
          Dag H. Wanvik made changes -
          Issue & fix info [Patch Available]
          Dag H. Wanvik made changes -
          Attachment derby-4477-partial-2.diff [ 12431167 ]
          Attachment derby-4477-partial-2.stat [ 12431168 ]
          Dag H. Wanvik made changes -
          Issue & fix info [Patch Available]
          Dag H. Wanvik made changes -
          Attachment derby-4477-lowmem.diff [ 12431679 ]
          Attachment derby-4477-lowmem.stat [ 12431680 ]
          Dag H. Wanvik made changes -
          Issue & fix info [Patch Available]
          Dag H. Wanvik made changes -
          Attachment derby-4477-lowmem-2.diff [ 12431774 ]
          Attachment derby-4477-lowmem-2.stat [ 12431775 ]
          Dag H. Wanvik made changes -
          Attachment derby-4477-lowmem-2.diff [ 12431774 ]
          Dag H. Wanvik made changes -
          Attachment derby-4477-lowmem-2.stat [ 12431775 ]
          Dag H. Wanvik made changes -
          Attachment derby-4477-lowmem-2.diff [ 12431778 ]
          Attachment derby-4477-lowmem-2.stat [ 12431779 ]
          Dag H. Wanvik made changes -
          Attachment derby-4477-lowmem-followup.diff [ 12434448 ]
          Attachment derby-4477-lowmem-followup.stat [ 12434449 ]
          Dag H. Wanvik made changes -
          Issue & fix info [Patch Available]
          Dag H. Wanvik made changes -
          Assignee Kristian Waagan [ kristwaa ] Dag H. Wanvik [ dagw ]
          Dag H. Wanvik made changes -
          Link This issue relates to DERBY-3645 [ DERBY-3645 ]
          Dag H. Wanvik made changes -
          Link This issue relates to DERBY-3646 [ DERBY-3646 ]
          Dag H. Wanvik made changes -
          Attachment derby-4477-useCloning.diff [ 12435448 ]
          Attachment derby-4477-useCloning.stat [ 12435449 ]
          Dag H. Wanvik made changes -
          Issue & fix info [Patch Available]
          Dag H. Wanvik made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Issue & fix info [Patch Available]
          Resolution Not A Problem [ 8 ]
          Dag H. Wanvik made changes -
          Resolution Not A Problem [ 8 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Dag H. Wanvik made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Fix Version/s 10.6.0.0 [ 12313727 ]
          Resolution Fixed [ 1 ]
          Dag H. Wanvik made changes -
          Component/s SQL [ 11408 ]
          Kristian Waagan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Kathey Marsden made changes -
          Link This issue is required by DERBY-4728 [ DERBY-4728 ]
          Kathey Marsden made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Kathey Marsden made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Fix Version/s 10.5.3.1 [ 12314182 ]
          Resolution Fixed [ 1 ]
          Gavin made changes -
          Workflow jira [ 12484737 ] Default workflow, editable Closed status [ 12800832 ]

            People

            • Assignee:
              Dag H. Wanvik
              Reporter:
              Kristian Waagan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development