Derby
  1. Derby
  2. DERBY-877

zOS - with DerbyClient getDate(#) fails with IllegalArgumentException - unsupported date format - resultset.java

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.1.1.0
    • Fix Version/s: 10.1.3.1, 10.2.1.6
    • Component/s: Network Client
    • Labels:
      None
    • Environment:
      OS/390 with classic (IBM) jvm142

      Description

      The test lang/resultset.java fails with DerbyNetClient on zOS
      because ResultSet.getDate(#) fails with an
      java.lang.IllegalArgumentException - unsupported date format.

      This is the stack trace with 10.2 debug version (but it fails
      with 10.1 also):
      ------------------
      ....
      getBytes(dt) got exception
      Data Conversion SQLException
      FAIL – unexpected exception:
      java.lang.IllegalArgumentException: Unsupported date format!
      java.lang.IllegalArgumentException: Unsupported date format!
      at
      org.apache.derby.client.am.DateTime.dateBytesToDate(DateTime.java:63)
      at
      org.apache.derby.client.am.Cursor.getDATE(Cursor.java:400)
      at
      org.apache.derby.client.am.Cursor.getDate(Cursor.java:712)
      at
      org.apache.derby.client.am.ResultSet.getDate(ResultSet.java:687)
      at
      org.apache.derbyTesting.functionTests.tests.jdbcapi.resultset.main(Unknown Source)
      ------------------
      Note: does not fail with jcc.

      Also, test lang/updatableResultSet.java failed with e.g.:

      • instead of 'Got expected exception : Illegal Conversion' :
        'Got expected exception : Unsupported date format!' .
      • instead of 'Got expected exception : Illegal Conversion' :
        'Got expected exception : nanos > 99999999999 or < 0' .
      1. derby-877.diff
        16 kB
        Kathey Marsden
      2. TestEnc.java
        7 kB
        Kathey Marsden

        Activity

        Hide
        Kathey Marsden added a comment -

        submitted this fix to the trunk and 10.1

        Show
        Kathey Marsden added a comment - submitted this fix to the trunk and 10.1
        Hide
        Daniel John Debrunner added a comment -

        Sounds good - fix the bug & register the cleanup issue.

        Show
        Daniel John Debrunner added a comment - Sounds good - fix the bug & register the cleanup issue.
        Hide
        Kathey Marsden added a comment -

        I think I will buy deceptive, but deceptive in the way that passing the encoding around the way client generally does in other places is deceptive. Also I think maybe the right thing is for these classes to actually go away moving forward.

        How about I submit this fix to provide the user some relief and then file another for the general encoding cleanup. I had planned to do that anyway based on some things I found while investigating this issue.

        Show
        Kathey Marsden added a comment - I think I will buy deceptive, but deceptive in the way that passing the encoding around the way client generally does in other places is deceptive. Also I think maybe the right thing is for these classes to actually go away moving forward. How about I submit this fix to provide the user some relief and then file another for the general encoding cleanup. I had planned to do that anyway based on some things I found while investigating this issue.
        Hide
        Daniel John Debrunner added a comment -

        This code looks wrong to me, at least it 's deceptive. Here's on example:

        + String timestamp = new String(buffer, offset,
        + DateTime.timestampRepresentationLength, encoding);

        If the encoding can be changed then the length of the byte representation should be changable.
        The length here is hard-coded as 26 using the constant timestampRepresentationLength.

        If really the server is always sending the data with a fixed encoding, which looks like the case, then I think the
        encoding should be fixed here. Otherwise it looks like the code can handle any encoding, and someone in the
        future may try to take advantage of that, which may or may not work in some cases and fail in others.

        Show
        Daniel John Debrunner added a comment - This code looks wrong to me, at least it 's deceptive. Here's on example: + String timestamp = new String(buffer, offset, + DateTime.timestampRepresentationLength, encoding); If the encoding can be changed then the length of the byte representation should be changable. The length here is hard-coded as 26 using the constant timestampRepresentationLength. If really the server is always sending the data with a fixed encoding, which looks like the case, then I think the encoding should be fixed here. Otherwise it looks like the code can handle any encoding, and someone in the future may try to take advantage of that, which may or may not work in some cases and fail in others.
        Hide
        Bryan Pendleton added a comment -

        Your change looks reasonable to me. I was able to reproduce the problem using your script, using Sun JDK 1.5 on Windows XP. I applied your patch to my code, and I was able to confirm that the problem went away and the StringOutOfBounds exceptions no longer occur.

        I'm no expert on encodings, but I can confirm that your repro program works for me and your patch corrects the problem for me.

        Show
        Bryan Pendleton added a comment - Your change looks reasonable to me. I was able to reproduce the problem using your script, using Sun JDK 1.5 on Windows XP. I applied your patch to my code, and I was able to confirm that the problem went away and the StringOutOfBounds exceptions no longer occur. I'm no expert on encodings, but I can confirm that your repro program works for me and your patch corrects the problem for me.
        Hide
        Kathey Marsden added a comment -

        I'll commit this patch tomorrow. I would appreciate a quick review from whomever is available.
        Even though the bug is marked zOS it is a generic issue that affects any jvm where the default jvm encoding for the characters in TIME/TIMESTAMP/DATE do not match network server.

        Thanks

        Kathey

        Show
        Kathey Marsden added a comment - I'll commit this patch tomorrow. I would appreciate a quick review from whomever is available. Even though the bug is marked zOS it is a generic issue that affects any jvm where the default jvm encoding for the characters in TIME/TIMESTAMP/DATE do not match network server. Thanks Kathey
        Hide
        Kathey Marsden added a comment -

        The patch fixes issues with getString, getTimeStamp, getDate and getTime on TIMESTAMP, DATE and TIME columns when the client JVM encoding does not match the server encoding for the characters being evaluated in DateTime.java methods

        • Changes the following methods in DateTime.java to take encoding parameter and create string based on encoding.
          dateBytesToDate, timeBytesToTime, timeBytesToTimeStamp, dateBytesToTimeStamp, timestampBytesToDate, timestampBytesToTime
        • Changes calling code to pass column encoding and throw SQLExceptions for UnsupportedEncoding exceptions if thrown from the methods above.

        Tests: derbyall passed as did the repro attached to this issue on Windows with Sun JDK 1.5 (note repro won't run with jdk 1.4.2).
        I still do not have a solution for testing within the harness on systems where the JVM encoding does match the server encoding.
        Any ideas on testing solutions are welcome.

        Show
        Kathey Marsden added a comment - The patch fixes issues with getString, getTimeStamp, getDate and getTime on TIMESTAMP, DATE and TIME columns when the client JVM encoding does not match the server encoding for the characters being evaluated in DateTime.java methods Changes the following methods in DateTime.java to take encoding parameter and create string based on encoding. dateBytesToDate, timeBytesToTime, timeBytesToTimeStamp, dateBytesToTimeStamp, timestampBytesToDate, timestampBytesToTime Changes calling code to pass column encoding and throw SQLExceptions for UnsupportedEncoding exceptions if thrown from the methods above. Tests: derbyall passed as did the repro attached to this issue on Windows with Sun JDK 1.5 (note repro won't run with jdk 1.4.2). I still do not have a solution for testing within the harness on systems where the JVM encoding does match the server encoding. Any ideas on testing solutions are welcome.
        Hide
        Kathey Marsden added a comment -

        Updated repro

        • Adds getString, getTimestamp, getTime and getDate calls for TIME/DATE/TIMESTAMP
        • Allows user to set output.encoding System property to allow more readable output on Windows and z/OS

        To repro on Windows, You must run with jdk 1.5
        java -Dfile.encoding=UTF-16 -Doutput.encoding=US-ASCII TestEnc
        COLUMN 1:TS TIMESTAMP
        java.lang.StringIndexOutOfBoundsException: String index out of range: 14
        at java.lang.String.charAt(String.java:558)
        at org.apache.derby.client.am.DateTime.timestampBytesToTimestamp(Unknown Source)
        at org.apache.derby.client.am.Cursor.getStringFromTIMESTAMP(Unknown Source)
        at org.apache.derby.client.am.Cursor.getString(Unknown Source)
        at org.apache.derby.client.am.ResultSet.getString(Unknown Source)
        at TestEnc.checkGetters(TestEnc.java:75)
        at TestEnc.go(TestEnc.java:49)
        at TestEnc.main(TestEnc.java:26)

        To repro on z/OS
        java -Doutput.encoding=Cp500 TestEnc

        Note this repro shows a bug that getTImetamp() on a TIME column will show date 1900-01-01-01.
        I will file a separate bug for that.

        Show
        Kathey Marsden added a comment - Updated repro Adds getString, getTimestamp, getTime and getDate calls for TIME/DATE/TIMESTAMP Allows user to set output.encoding System property to allow more readable output on Windows and z/OS To repro on Windows, You must run with jdk 1.5 java -Dfile.encoding=UTF-16 -Doutput.encoding=US-ASCII TestEnc COLUMN 1:TS TIMESTAMP java.lang.StringIndexOutOfBoundsException: String index out of range: 14 at java.lang.String.charAt(String.java:558) at org.apache.derby.client.am.DateTime.timestampBytesToTimestamp(Unknown Source) at org.apache.derby.client.am.Cursor.getStringFromTIMESTAMP(Unknown Source) at org.apache.derby.client.am.Cursor.getString(Unknown Source) at org.apache.derby.client.am.ResultSet.getString(Unknown Source) at TestEnc.checkGetters(TestEnc.java:75) at TestEnc.go(TestEnc.java:49) at TestEnc.main(TestEnc.java:26) To repro on z/OS java -Doutput.encoding=Cp500 TestEnc Note this repro shows a bug that getTImetamp() on a TIME column will show date 1900-01-01-01. I will file a separate bug for that.
        Hide
        Kathey Marsden added a comment -

        Please ignore previously deleted comment in favor of this one.

        Here is a summary if this issue. I will submit a patch next week. Please read and comment if you have time. To repro on windows with jdk 1.5:

        java -Dfile.encoding=UTF-16 -DConsole.encoding=Cp1252 TestEnc

        ISSUE:
        Encoding issues with getDate(), getTime, and getTimeStamp() when the JVM encoding does not match server encoding. Failures are manifested with various exceptions indicating the data is not formatted properly.

        BACKGROUND:
        In DRDA, the rule for data encoding for result set data values is that the receiver makes right. So if the server sends a column with Ccsid 1208 (UTF-8), the client is responsible for handling that format correctly. Alternately if the client sends a parameter, the server is responsible for handling the encoding properly.

        Network Server always sends data in UTF-8 (CCSID- 1208) format, so the client really only needs to handle that for result set data.

        Note: DDM String values are always EBCDIC (but that seems to be handled properly)

        PROBLEM:

        Methods in org.apache.derby.client.am.DateTime.java make the assumption that the default encoding of the JVM matches the encoding of the date/time/timestamp value for the characters sent in those values. For example, the method timeStampBytesToTimestamp takes a byte[] buffer as sent by the server and returns a java.sql.Timestamp, computing values like this:

        String date = new String(buffer, offset, DateTime.dateRepresentationLength);
        [snip]
        int zeroBase = ((int) '0');
        year =
        1000 * (((int) date.charAt(yearIndx)) - zeroBase) + ...

        When the default encoding for the JVM is not compatible with the encoding sent by Network Server, (for example Cp500 or Cp037 are EBCDIC), zero base is caluculated incorectly, thus the resulting values are garbage.

        Other methods like timestampToTimestampBytes go the exact reverse. They build up a character array and then turn it into a String and do getBytes() without specifying the encoding. It seems that these would affect setTimestamp/setTime/setDate calls, but I will file this as a separate issue once I have a repro.

        One other issue regarding DateTime.java is that it uses deprecated methods such as Date.setYear() deprecated since jdk 1.1.

        FIX
        As a fix for 10.1 I am thinking the solution is to pass the encoding of the column into the methods in DateTime.java that create Strings and then create the string with the column encoding.
        e.g
        timeStampBytesToTimestamp will have
        String date = new String(buffer, offset, DateTime.dateRepresentationLength,encoding);
        Then the math will be correct.

        I think longer term we should reconsider DateTime.java altogether and consider how much all of this character arithmetic is buying us, especially give it is relying on deprecated methods etc.

        TESTING
        I can use Deepa's trick with jdk 1.5 to reproduce the issue, but hope we can find a more generic way to test these encoding issues so that they can be encluded in the nightly regressions.

        java -Dfile.encoding=UTF-16 -DConsole.encoding=Cp1252 TestEnc

        OTHER AREAS OF CONCERN
        I looked briefly at the code for additional potential encoding problems. I saw these possible issues.

        DateTime.java
        Need to look into setTime/setDate/setTimestamp cases. I have not seen errors so maybe they are ok.

        Decimal.java, FloatingPoint.java
        These converters need closer examination.

        Clob.java
        Has an encoding_ field but it looks like it is not used.

        NetConnectionReply.java
        has a getBytes() call that needs consideration.

        Cursor.java

        In getVARCHAR there is code like this that I don't quite understand. If the server uses Ccsid 1208, why do we call getStringWithoutConvert for 1200?

        if(ccsid_[column-1] == 1200)
        return getStringWithoutConvert (columnDataPosition_[column-1] + 2, columnDataComputedLength_[column-1] - 2, fdocaLength_[column-1]);

        Show
        Kathey Marsden added a comment - Please ignore previously deleted comment in favor of this one. Here is a summary if this issue. I will submit a patch next week. Please read and comment if you have time. To repro on windows with jdk 1.5: java -Dfile.encoding=UTF-16 -DConsole.encoding=Cp1252 TestEnc ISSUE: Encoding issues with getDate(), getTime, and getTimeStamp() when the JVM encoding does not match server encoding. Failures are manifested with various exceptions indicating the data is not formatted properly. BACKGROUND: In DRDA, the rule for data encoding for result set data values is that the receiver makes right. So if the server sends a column with Ccsid 1208 (UTF-8), the client is responsible for handling that format correctly. Alternately if the client sends a parameter, the server is responsible for handling the encoding properly. Network Server always sends data in UTF-8 (CCSID- 1208) format, so the client really only needs to handle that for result set data. Note: DDM String values are always EBCDIC (but that seems to be handled properly) PROBLEM: Methods in org.apache.derby.client.am.DateTime.java make the assumption that the default encoding of the JVM matches the encoding of the date/time/timestamp value for the characters sent in those values. For example, the method timeStampBytesToTimestamp takes a byte[] buffer as sent by the server and returns a java.sql.Timestamp, computing values like this: String date = new String(buffer, offset, DateTime.dateRepresentationLength); [snip] int zeroBase = ((int) '0'); year = 1000 * (((int) date.charAt(yearIndx)) - zeroBase) + ... When the default encoding for the JVM is not compatible with the encoding sent by Network Server, (for example Cp500 or Cp037 are EBCDIC), zero base is caluculated incorectly, thus the resulting values are garbage. Other methods like timestampToTimestampBytes go the exact reverse. They build up a character array and then turn it into a String and do getBytes() without specifying the encoding. It seems that these would affect setTimestamp/setTime/setDate calls, but I will file this as a separate issue once I have a repro. One other issue regarding DateTime.java is that it uses deprecated methods such as Date.setYear() deprecated since jdk 1.1. FIX As a fix for 10.1 I am thinking the solution is to pass the encoding of the column into the methods in DateTime.java that create Strings and then create the string with the column encoding. e.g timeStampBytesToTimestamp will have String date = new String(buffer, offset, DateTime.dateRepresentationLength,encoding); Then the math will be correct. I think longer term we should reconsider DateTime.java altogether and consider how much all of this character arithmetic is buying us, especially give it is relying on deprecated methods etc. TESTING I can use Deepa's trick with jdk 1.5 to reproduce the issue, but hope we can find a more generic way to test these encoding issues so that they can be encluded in the nightly regressions. java -Dfile.encoding=UTF-16 -DConsole.encoding=Cp1252 TestEnc OTHER AREAS OF CONCERN I looked briefly at the code for additional potential encoding problems. I saw these possible issues. DateTime.java Need to look into setTime/setDate/setTimestamp cases. I have not seen errors so maybe they are ok. Decimal.java, FloatingPoint.java These converters need closer examination. Clob.java Has an encoding_ field but it looks like it is not used. NetConnectionReply.java has a getBytes() call that needs consideration. Cursor.java In getVARCHAR there is code like this that I don't quite understand. If the server uses Ccsid 1208, why do we call getStringWithoutConvert for 1200? if(ccsid_ [column-1] == 1200) return getStringWithoutConvert (columnDataPosition_ [column-1] + 2, columnDataComputedLength_ [column-1] - 2, fdocaLength_ [column-1] );

          People

          • Assignee:
            Kathey Marsden
            Reporter:
            Myrna van Lunteren
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development