Derby
  1. Derby
  2. DERBY-4584

Unable to connect to network server if client thread name has Japanese characters

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.2.1, 10.1.3.1, 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0
    • Fix Version/s: 10.5.3.1, 10.6.1.0
    • Component/s: Network Client
    • Labels:
      None

      Description

      I am opening this bug, which is probably a duplicate of bug#728 so that other's may find it and save the hours I spent chasing it down. Feel free to mark this as a duplicate. However, while related, it may not be a true duplicate of 728.

      The exception is similar to 728:

      Exception in thread "main" org.apache.derby.client.am.SqlException: Unicode string can't convert to Ebcdic string

      (Here is the version of the exception I received – excuse the Japanese characters):

      Caused by: org.apache.derby.client.am.SqlException: Unicode ストリングを EBCDIC ストリングに変換することはできません。
      at org.apache.derby.client.net.EbcdicCcsidManager.convertFromUCS2(Unknown Source)
      at org.apache.derby.client.net.Request.writeScalarString(Unknown Source)
      at org.apache.derby.client.net.Request.writeScalarString(Unknown Source)
      at org.apache.derby.client.net.NetConnectionRequest.buildEXTNAM(Unknown Source)
      at org.apache.derby.client.net.NetConnectionRequest.buildEXCSAT(Unknown Source)
      at org.apache.derby.client.net.NetConnectionRequest.writeExchangeServerAttributes(Unknown Source)
      at org.apache.derby.client.net.NetConnection.writeServerAttributesAndKeyExchange(Unknown Source)
      at org.apache.derby.client.net.NetConnection.flowServerAttributesAndKeyExchange(Unknown Source)
      at org.apache.derby.client.net.NetConnection.flowUSRIDPWDconnect(Unknown Source)
      at org.apache.derby.client.net.NetConnection.flowConnect(Unknown Source)
      at org.apache.derby.client.net.NetConnection.initialize(Unknown Source)
      at org.apache.derby.client.net.NetConnection.<init>(Unknown Source)
      at org.apache.derby.client.net.NetConnection40.<init>(Unknown Source)
      at org.apache.derby.client.net.ClientJDBCObjectFactoryImpl40.newNetConnection(Unknown Source)
      at org.apache.derby.client.net.NetXAConnection.createNetConnection(Unknown Source)
      at org.apache.derby.client.net.NetXAConnection.<init>(Unknown Source)
      at org.apache.derby.client.ClientPooledConnection.getNetXAConnection(Unknown Source)
      ... 45 more

      However, the difference is that the database name (and connection URL) does NOT contain unicode characters. In this case, the thread name contains Japanese characters. If the thread performing java.sql.DriverManager.getConnection() has characters that cannot be translated into EBCDIC the above exception is the result. If the thread name is changed to contain only standard ASCII characters, the connection to the DB is successful. Note again, in my case, the connection URL is a standard connection URL with no i18n characters, something similar to:

      jdbc:derby://localhost/database

      It is only the thread-name that contains i18n characters. I don't know why the client feels it necessary to marshall the client-thread name, but that seems to be the problem. The fix for this issue is likely easier than 728 if the requirement that the client marshall the thread name can be removed (it seems senseless).

      Finally, just for the record, a typical thread name that tickles this bug is:

      "Running-2 (MOTDバナーの設定 for 10.0.0.90@Default)"

      If the Japanese is removed from the thread names, there is no problem.

      The workaround in our case was to change the thread names in our code to not contain Japanese characters.

      1. DERBY-4584.diff
        1 kB
        Tiago R. Espinha
      2. DERBY-4584-test.diff
        1.0 kB
        Tiago R. Espinha
      3. DERBY-4584-fix.diff
        0.8 kB
        Tiago R. Espinha
      4. DERBY-4584-fix.diff
        0.9 kB
        Tiago R. Espinha
      5. DERBY-4584-test.diff
        1 kB
        Tiago R. Espinha

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          14d 4h 41m 1 Knut Anders Hatlen 30/Mar/10 12:24
          Closed Closed Reopened Reopened
          84d 9h 20m 1 Kathey Marsden 01/Jul/10 22:56
          Reopened Reopened Resolved Resolved
          19d 20h 2m 1 Mike Matrigali 21/Jul/10 18:59
          Resolved Resolved Closed Closed
          84d 17h 2 Tiago R. Espinha 05/Oct/10 10:47
          Gavin made changes -
          Workflow jira [ 12501936 ] Default workflow, editable Closed status [ 12800434 ]
          Tiago R. Espinha made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Tiago R. Espinha added a comment -

          Closing this issue. Since this issue is related to Japanese characters on the EXCSAT command (which must be negotiated in EBCDIC), there's nothing DERBY-728 can do here other than convert the characters into question marks like it has been done.

          Show
          Tiago R. Espinha added a comment - Closing this issue. Since this issue is related to Japanese characters on the EXCSAT command (which must be negotiated in EBCDIC), there's nothing DERBY-728 can do here other than convert the characters into question marks like it has been done.
          Mike Matrigali made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Assignee Mike Matrigali [ mikem ] Tiago R. Espinha [ espinha ]
          Fix Version/s 10.5.3.1 [ 12314182 ]
          Resolution Fixed [ 1 ]
          Hide
          Mike Matrigali added a comment -

          backported fix to 10.5, resolving as fixed and resetting original owner.

          Show
          Mike Matrigali added a comment - backported fix to 10.5, resolving as fixed and resetting original owner.
          Hide
          Mike Matrigali added a comment -

          i am looking at backporting this issue to 10.5.

          Show
          Mike Matrigali added a comment - i am looking at backporting this issue to 10.5.
          Mike Matrigali made changes -
          Assignee Tiago R. Espinha [ espinha ] Mike Matrigali [ mikem ]
          Kathey Marsden made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Kathey Marsden made changes -
          Link This issue is required by DERBY-4728 [ DERBY-4728 ]
          Tiago R. Espinha made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Tiago R. Espinha added a comment -

          Regressions look fine, so I'm closing the issue.

          Show
          Tiago R. Espinha added a comment - Regressions look fine, so I'm closing the issue.
          Hide
          Tiago R. Espinha added a comment -

          I was more concerned about this:

          java.lang.Exception: DRDA_InvalidReplyHeader2.S:Invalid reply header from network server: Invalid string ?. Plaintext connection attempt to an SSL enabled server?

          It's complaining about an invalid string composed of a question mark... but we'll wait to see tomorrow's regressions.

          Show
          Tiago R. Espinha added a comment - I was more concerned about this: java.lang.Exception: DRDA_InvalidReplyHeader2.S:Invalid reply header from network server: Invalid string ?. Plaintext connection attempt to an SSL enabled server? It's complaining about an invalid string composed of a question mark... but we'll wait to see tomorrow's regressions.
          Hide
          Knut Anders Hatlen added a comment -

          Looks like it only happened on one of the machines, so let's wait and see if it happens again or if it's just a glitch. It sounds unlikely that the changes made here should be preventing the network server from starting up.

          Show
          Knut Anders Hatlen added a comment - Looks like it only happened on one of the machines, so let's wait and see if it happens again or if it's just a glitch. It sounds unlikely that the changes made here should be preventing the network server from starting up.
          Hide
          Tiago R. Espinha added a comment -

          Knut,

          I was looking at today's regression testing and this came up:

          http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.4/testing/testlog/lin/929180-suitesAll_diff.txt

          It only seems to happen in JVM 1.4 though... do you think this is related to my change?

          Show
          Tiago R. Espinha added a comment - Knut, I was looking at today's regression testing and this came up: http://dbtg.foundry.sun.com/derby/test/Daily/jvm1.4/testing/testlog/lin/929180-suitesAll_diff.txt It only seems to happen in JVM 1.4 though... do you think this is related to my change?
          Knut Anders Hatlen made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 10.6.0.0 [ 12313727 ]
          Resolution Fixed [ 1 ]
          Hide
          Knut Anders Hatlen added a comment -

          Thanks, Tiago, for the new patch. It looks fine, and all the regression tests ran cleanly. Committed revision 929085.

          Show
          Knut Anders Hatlen added a comment - Thanks, Tiago, for the new patch. It looks fine, and all the regression tests ran cleanly. Committed revision 929085.
          Hide
          Tiago R. Espinha added a comment -

          suites.All ran without problems (just the "normal" replication test failure since I'm using a Windows box)

          Next, I'll be running derbyall.

          Show
          Tiago R. Espinha added a comment - suites.All ran without problems (just the "normal" replication test failure since I'm using a Windows box) Next, I'll be running derbyall.
          Tiago R. Espinha made changes -
          Attachment DERBY-4584-fix.diff [ 12440125 ]
          Attachment DERBY-4584-test.diff [ 12440126 ]
          Hide
          Tiago R. Espinha added a comment -

          Thank you for your comments Knut. I'm uploading two new patches and I'll be running suites.All now.

          Show
          Tiago R. Espinha added a comment - Thank you for your comments Knut. I'm uploading two new patches and I'll be running suites.All now.
          Hide
          Knut Anders Hatlen added a comment -

          Thanks, the fix looks fine to me, but I think we need to add comments to the code both in NetConnection and in the test, as the purpose of this code is not obvious just by looking at it. Also, I think the connection returned by BaseJDBCTestCase.openConnection(String) is not closed automatically by the test harness, so it needs to be closed manually in the test case. DRDAProtocolTest.tearDown() also needs to call super.tearDown().

          Show
          Knut Anders Hatlen added a comment - Thanks, the fix looks fine to me, but I think we need to add comments to the code both in NetConnection and in the test, as the purpose of this code is not obvious just by looking at it. Also, I think the connection returned by BaseJDBCTestCase.openConnection(String) is not closed automatically by the test harness, so it needs to be closed manually in the test case. DRDAProtocolTest.tearDown() also needs to call super.tearDown().
          Tiago R. Espinha made changes -
          Attachment DERBY-4584-test.diff [ 12440063 ]
          Attachment DERBY-4584-fix.diff [ 12440064 ]
          Hide
          Tiago R. Espinha added a comment -

          That does the trick Knut.

          I'm submitting two files now:

          • DERBY-4584-test.diff
            This adds a fixture to DRDAProtocolTest that makes the exception pop. For this test I'm using the Chinese character in DERBY-728 but it is equally good as a Japanese test for this purpose.
          • DERBY-4584-fix.diff
            This implements the fix as suggested and the fixture mentioned above no longer fails.
          Show
          Tiago R. Espinha added a comment - That does the trick Knut. I'm submitting two files now: DERBY-4584 -test.diff This adds a fixture to DRDAProtocolTest that makes the exception pop. For this test I'm using the Chinese character in DERBY-728 but it is equally good as a Japanese test for this purpose. DERBY-4584 -fix.diff This implements the fix as suggested and the fixture mentioned above no longer fails.
          Hide
          Knut Anders Hatlen added a comment -

          Looks like you're right, extnam_ is used and converted to EBCDIC more places than constructPrddta(), so it's probably best to do this in constructExtnam(). As far as I understand, we are able to convert all characters whose Unicode codepoint is less than 256 to EBCDIC, so perhaps something like this would do?

          char[] chars = threadName.toCharArray();
          for (int i = 0; i < chars.length; i++)

          { if (chars[i] > 0xff) chars[i] = '?'; }

          threadName = new String(chars);

          Show
          Knut Anders Hatlen added a comment - Looks like you're right, extnam_ is used and converted to EBCDIC more places than constructPrddta(), so it's probably best to do this in constructExtnam(). As far as I understand, we are able to convert all characters whose Unicode codepoint is less than 256 to EBCDIC, so perhaps something like this would do? char[] chars = threadName.toCharArray(); for (int i = 0; i < chars.length; i++) { if (chars[i] > 0xff) chars[i] = '?'; } threadName = new String(chars);
          Hide
          Tiago R. Espinha added a comment -

          Hello Knut,

          Thank you for your comments; I overlooked the fact that the behavior is unspecified but indeed, then we cannot use it. All things considered we might just use the method you suggested instead.

          Maybe I can create an overload of convertFromUCS2() with an encodeExtraCharacters flag where I implement your suggestion, and then the existing method invokes that one with this flag set to false. This way we maintain the existing behavior for compatibility and we have a way to encode UCS2 strings in EBCDIC whenever the content of the extra characters isn't important.

          You suggested though that we do this on the constructPrddta method, but is it wise to simply convert the EXTNAM on the fly rather than converting it in a definite manner when it is constructed? (constructExtnam())

          If we simply convert it on the fly, then we must always be careful whenever we use the extnam_ as it will still contain UCS2 characters.

          I'll also look into setting the name of threads like you mentioned; if this is possible, I'll also add a fixture to test this behavior.

          Show
          Tiago R. Espinha added a comment - Hello Knut, Thank you for your comments; I overlooked the fact that the behavior is unspecified but indeed, then we cannot use it. All things considered we might just use the method you suggested instead. Maybe I can create an overload of convertFromUCS2() with an encodeExtraCharacters flag where I implement your suggestion, and then the existing method invokes that one with this flag set to false. This way we maintain the existing behavior for compatibility and we have a way to encode UCS2 strings in EBCDIC whenever the content of the extra characters isn't important. You suggested though that we do this on the constructPrddta method, but is it wise to simply convert the EXTNAM on the fly rather than converting it in a definite manner when it is constructed? (constructExtnam()) If we simply convert it on the fly, then we must always be careful whenever we use the extnam_ as it will still contain UCS2 characters. I'll also look into setting the name of threads like you mentioned; if this is possible, I'll also add a fixture to test this behavior.
          Hide
          Knut Anders Hatlen added a comment -

          As for testing, could we set the name of the JUnit execution thread with Thread.currentThread().setName("<string with Japanese characters>") in setUp() and reset it in tearDown()?

          Show
          Knut Anders Hatlen added a comment - As for testing, could we set the name of the JUnit execution thread with Thread.currentThread().setName("<string with Japanese characters>") in setUp() and reset it in tearDown()?
          Hide
          Knut Anders Hatlen added a comment -

          Hi Tiago,

          Thanks for the patch. I think there are some problems with this approach:

          1) The javadoc for java.lang.String#getBytes(java.lang.String) says that the behaviour is unspecified if the string cannot be encoded in the given charset. So we cannot rely on that method to replace non-ascii characters with question marks.

          2) The String constructor should also take the charset name as an argument so that the code works regardless of what the system default encoding is.

          3) We should avoid printing exceptions on the console. I think it would be better instead to do a "throw SqlException.javaException(agent_.logWriter, e)".

          4) EBCDIC supports more characters than US-ASCII does, so this approach will also replace some valid EBCDIC characters with question marks.

          I noticed that EbcdicCcsidManager.convertFromUCS2() has commented out code to generate a question mark instead of raising an exception for multi-byte characters. Would it be possible to add a flag to re-enable that code and call the method with that flag when converting extnam_ to EBCDIC in NetConnection.constructPrddta()?

          Show
          Knut Anders Hatlen added a comment - Hi Tiago, Thanks for the patch. I think there are some problems with this approach: 1) The javadoc for java.lang.String#getBytes(java.lang.String) says that the behaviour is unspecified if the string cannot be encoded in the given charset. So we cannot rely on that method to replace non-ascii characters with question marks. 2) The String constructor should also take the charset name as an argument so that the code works regardless of what the system default encoding is. 3) We should avoid printing exceptions on the console. I think it would be better instead to do a "throw SqlException.javaException(agent_.logWriter, e)". 4) EBCDIC supports more characters than US-ASCII does, so this approach will also replace some valid EBCDIC characters with question marks. I noticed that EbcdicCcsidManager.convertFromUCS2() has commented out code to generate a question mark instead of raising an exception for multi-byte characters. Would it be possible to add a flag to re-enable that code and call the method with that flag when converting extnam_ to EBCDIC in NetConnection.constructPrddta()?
          Tiago R. Espinha made changes -
          Attachment DERBY-4584.diff [ 12440019 ]
          Hide
          Tiago R. Espinha added a comment -

          This patch uses Kathey's suggestion for the thread name.

          The problem now remains as to how will we test this patch. Is there any way to get our thread names to show in Japanese?

          Show
          Tiago R. Espinha added a comment - This patch uses Kathey's suggestion for the thread name. The problem now remains as to how will we test this patch. Is there any way to get our thread names to show in Japanese?
          Tiago R. Espinha made changes -
          Assignee Tiago R. Espinha [ espinha ]
          Hide
          Kathey Marsden added a comment -

          As I recall EXTNAM is part of the EXTSAT command which will still be in EBCDIC, even with the new ACR https://issues.apache.org/jira/secure/attachment/12405588/ACR7007.pdf

          So, I think maybe mapping multibyte characters to question marks or some such will be how we will need to handle this.

          Show
          Kathey Marsden added a comment - As I recall EXTNAM is part of the EXTSAT command which will still be in EBCDIC, even with the new ACR https://issues.apache.org/jira/secure/attachment/12405588/ACR7007.pdf So, I think maybe mapping multibyte characters to question marks or some such will be how we will need to handle this.
          Hide
          Dag H. Wanvik added a comment -

          This code in org.apache.derby.client.net.NetConnection
          is the source of this, presumably:

          private void constructExtnam() throws SqlException

          { extnam_ = "derbydnc" + java.lang.Thread.currentThread().getName(); }

          This is a quantity of the DRDA protocol: (quote)
          DRDA, Version 3, Volume 1, section 4.4.1 Accessing a Remote Relational Database Manager

          "The extnam is the name of a job, task, or process that the application requester services.
          It is used for diagnostic/logging purposes. In this example, it is the name of the job that
          contains the execution of the application that is invoking application requester
          functions on the OS/400 system.
          Note: This parameter is required and must contain the name of the application requester's
          execution thread in its operating environment. It must be a name that an observer of
          the operating environment can easily associate with its execution."

          I think there is a new version of DRDA coming that would allow Unicode in such strings; and if/when we move to that, it might solve this issue. A more immediate way to this is to algoritmically modify a non-EBCDIC thread name into something EBCDIC...

          Show
          Dag H. Wanvik added a comment - This code in org.apache.derby.client.net.NetConnection is the source of this, presumably: private void constructExtnam() throws SqlException { extnam_ = "derbydnc" + java.lang.Thread.currentThread().getName(); } This is a quantity of the DRDA protocol: (quote) DRDA, Version 3, Volume 1, section 4.4.1 Accessing a Remote Relational Database Manager "The extnam is the name of a job, task, or process that the application requester services. It is used for diagnostic/logging purposes. In this example, it is the name of the job that contains the execution of the application that is invoking application requester functions on the OS/400 system. Note: This parameter is required and must contain the name of the application requester's execution thread in its operating environment. It must be a name that an observer of the operating environment can easily associate with its execution." I think there is a new version of DRDA coming that would allow Unicode in such strings; and if/when we move to that, it might solve this issue. A more immediate way to this is to algoritmically modify a non-EBCDIC thread name into something EBCDIC...
          Hide
          Kathey Marsden added a comment -

          Thanks for filing this issue.
          Maybe we use the thread name in something we send in the protocol, but I am not sure why we would. I am also not sure if it would be fixed with the proposed changes for DERBY-728 or if this will need a separate fix. Linking to that issue so we make sure it is considered with that change.

          Show
          Kathey Marsden added a comment - Thanks for filing this issue. Maybe we use the thread name in something we send in the protocol, but I am not sure why we would. I am also not sure if it would be fixed with the proposed changes for DERBY-728 or if this will need a separate fix. Linking to that issue so we make sure it is considered with that change.
          Kathey Marsden made changes -
          Field Original Value New Value
          Link This issue relates to DERBY-728 [ DERBY-728 ]
          Brett Wooldridge created issue -

            People

            • Assignee:
              Tiago R. Espinha
              Reporter:
              Brett Wooldridge
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development