Derby
  1. Derby
  2. DERBY-1256

Remove usage of non-portable methods in derby code

    Details

    • Urgency:
      Normal
    • Issue & fix info:
      High Value Fix

      Description

      Remove usage of non portable methods that use native encoding as they could potentially lead to bugs on platforms with different encodings.

      Replace with code using fixed conversion, or alternative mechanisms.
      If the call is required its use should be commented as to why it is required.

      I went through the classes in java.io package in jdk1.4.2 and have come up with a list of methods, constructors in the java.io.* classes that use native encoding and did a java search on eclipse for the code in java/engine, java/testing, java/drda, java/client.

      1)ByteArrayOutputStream.toString()
      testing - DerbyNetNewServer:
      System.out.println( bos.toString());
      server- NetworkServerCtrlInfo.getCLSSysInfo, getNetSysInfo,

      2)DataInputStream.readLine()
      ArrayInputStream - org.apache.derby.iapi.services.io - java/engine
      readLine()
      CorruptRandomAccessFile - org.apache.derbyTesting.functionTests.util.corruptio - java/testing
      readLine()

      3)InputStreamReader(InputStream)
      StatementDuration - org.apache.derby.diag - java/engine -
      next()
      ImportReadData - org.apache.derby.impl.load - java/engine -
      realOpenFile()
      ErrorLogReader - org.apache.derby.diag - java/engine -
      next()
      UCode_CharStream - org.apache.derby.impl.sql.compile - java/engine -
      ReInit(InputStream, int, int, int)
      xmlBinding - org.apache.derbyTesting.functionTests.tests.lang - java/testing -
      insertDocWithDTD(Connection, String, String, String, int)
      BaseMonitor - org.apache.derby.impl.services.monitor - java/engine -
      dumpTempWriter(boolean)
      DbFile - org.apache.derbyTesting.functionTests.util - java/testing -
      stringFromFile(InputStream)
      HandleResult - org.apache.derbyTesting.functionTests.harness - java/testing -
      handleResult(int, InputStream, InputStream, PrintWriter, String) (2 matches)
      ProcessStreamResult - org.apache.derbyTesting.functionTests.harness - java/testing -
      run()
      FileCompare - org.apache.derbyTesting.functionTests.harness - java/testing -
      doSysDiff(InputStream, String, String, File, PrintWriter)
      exec(String, File, PrintWriter, String, String, String, int, boolean, boolean, String, String, String)
      UCode_CharStream(InputStream, int, int, int)
      insertFiles(Connection, String, String, int) (2 matches)

      4) OuputStreamWriter(OutputStream)

      ExportWriteData - org.apache.derby.impl.load - java/engine
      openFile()
      ProcessStreamResult - org.apache.derbyTesting.functionTests.harness - java/testing
      ProcessStreamResult(InputStream, BufferedOutputStream, String, String)
      RawStore - org.apache.derby.impl.store.raw - java/engine
      run()

      5) RandomAccessFile.readLine()
      ArrayInputStream - org.apache.derby.iapi.services.io - java/engine -
      readLine()
      CorruptRandomAccessFile - org.apache.derbyTesting.functionTests.util.corruptio - java/testing -
      readLine()
      DerbyNetAutoStart - org.apache.derbyTesting.functionTests.tests.derbynet - java/testing -
      checkLog(RandomAccessFile, String[])

      PrintStream and PrintWriter , print methods uses platforms default charset encoding.
      =============
      There are also non portable methods in String class
      new String(byte[])
      new String(byte[],int,int)
      String.getBytes(). There are jira entries already for these- DERBY-900, DERBY-901,DERBY-902,DERBY-903.

        Issue Links

          Activity

          Hide
          Knut Anders Hatlen added a comment -

          [bulk update] Close all resolved issues that haven't been updated for more than one year.

          Show
          Knut Anders Hatlen added a comment - [bulk update] Close all resolved issues that haven't been updated for more than one year.
          Hide
          Kathey Marsden added a comment -

          According to comments all known non-portable method issues have been addressed in other issues over various versions. Other specific problems should be logged as individual issues as found

          Show
          Kathey Marsden added a comment - According to comments all known non-portable method issues have been addressed in other issues over various versions. Other specific problems should be logged as individual issues as found
          Hide
          Myrna van Lunteren added a comment -

          That's good to know. I think mostly the theoretically non-portable methods left in the code are ok. However, it would be good if the ok-ed usage could be documented somehow...

          Show
          Myrna van Lunteren added a comment - That's good to know. I think mostly the theoretically non-portable methods left in the code are ok. However, it would be good if the ok-ed usage could be documented somehow...
          Hide
          Knut Anders Hatlen added a comment -

          I see a similar IOException when I run ReplicationRun_Local with file.encoding=UTF-16 on OpenSolaris/JDK6:

          java.io.IOException: Cannot run program "/usr/jdk/instances/jdk1.6.0/jre/bin/java": error=2, No such file or directory
          at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
          at java.lang.Runtime.exec(Runtime.java:593)
          at java.lang.Runtime.exec(Runtime.java:431)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun$4.run(ReplicationRun.java:2366)
          at java.lang.Thread.run(Thread.java:619)
          Caused by: java.io.IOException: error=2, No such file or directory
          at java.lang.UNIXProcess.forkAndExec(Native Method)
          at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
          at java.lang.ProcessImpl.start(ProcessImpl.java:65)
          at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
          ... 4 more

          However, I see the same problem when I execute the following code with file.encoding set (works fine without the property):

          public class FileEncoding {
          public static void main(String[] args) throws Exception

          { Process p = Runtime.getRuntime().exec("/usr/bin/ls"); System.out.println("Status: " + p.waitFor()); }

          }

          Exception in thread "main" java.io.IOException: Cannot run program "/usr/bin/ls": error=2, No such file or directory
          at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
          at java.lang.Runtime.exec(Runtime.java:593)
          at java.lang.Runtime.exec(Runtime.java:431)
          at java.lang.Runtime.exec(Runtime.java:328)
          at FileEncoding.main(FileEncoding.java:3)
          Caused by: java.io.IOException: error=2, No such file or directory
          at java.lang.UNIXProcess.forkAndExec(Native Method)
          at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
          at java.lang.ProcessImpl.start(ProcessImpl.java:65)
          at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
          ... 4 more

          So I don't think that it shows a problem in Derby or the tests.

          Show
          Knut Anders Hatlen added a comment - I see a similar IOException when I run ReplicationRun_Local with file.encoding=UTF-16 on OpenSolaris/JDK6: java.io.IOException: Cannot run program "/usr/jdk/instances/jdk1.6.0/jre/bin/java": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at java.lang.Runtime.exec(Runtime.java:593) at java.lang.Runtime.exec(Runtime.java:431) at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun$4.run(ReplicationRun.java:2366) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:53) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 4 more However, I see the same problem when I execute the following code with file.encoding set (works fine without the property): public class FileEncoding { public static void main(String[] args) throws Exception { Process p = Runtime.getRuntime().exec("/usr/bin/ls"); System.out.println("Status: " + p.waitFor()); } } Exception in thread "main" java.io.IOException: Cannot run program "/usr/bin/ls": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at java.lang.Runtime.exec(Runtime.java:593) at java.lang.Runtime.exec(Runtime.java:431) at java.lang.Runtime.exec(Runtime.java:328) at FileEncoding.main(FileEncoding.java:3) Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:53) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 4 more So I don't think that it shows a problem in Derby or the tests.
          Hide
          Knut Anders Hatlen added a comment -

          The use of InputStreamReader(InputStream) in ReplicationRun looks safe to me. The stream returned by Process.getInputStream() will use the default encoding of the platform, so the reader that decodes it should also use the default encoding.

          When you tested with file.encoding=UTF-16, did you make sure that the subprocesses also had that property set? If the subprocesses had different encoding than the main process, I guess there could be inconsistencies when decoding the output (though decoding the process output does not seem to be the problem, judging by the stack trace).

          Note also that the file.encoding property is not supposed to be modified, so any problems seen while running with that property set can just as well be in the JDK as in Derby. See for instance the evaluation comments in this bug report: http://bugs.sun.com/view_bug.do?bug_id=4163515

          Show
          Knut Anders Hatlen added a comment - The use of InputStreamReader(InputStream) in ReplicationRun looks safe to me. The stream returned by Process.getInputStream() will use the default encoding of the platform, so the reader that decodes it should also use the default encoding. When you tested with file.encoding=UTF-16, did you make sure that the subprocesses also had that property set? If the subprocesses had different encoding than the main process, I guess there could be inconsistencies when decoding the output (though decoding the process output does not seem to be the problem, judging by the stack trace). Note also that the file.encoding property is not supposed to be modified, so any problems seen while running with that property set can just as well be in the JDK as in Derby. See for instance the evaluation comments in this bug report: http://bugs.sun.com/view_bug.do?bug_id=4163515
          Hide
          Myrna van Lunteren added a comment -

          Manjula's test run (see DERBY-4019) suggests that the run is ok...

          However, I did some experiments with running the ReplicationRun_Local test with the property -Dfile.encoding="UTF-16", which I think only works with (sun) jdks 1.5 and up, and should also test encoding-safe usage. Running that test like that with jdk 1.5 passed consistenly on windows, but consistently failed on linux:
          ------------------------
          1) testReplication_Local_existingTestsAsReplLoad(org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun_Local)java.lang.Exception: Could not ping in 150 * 500ms.: DRDA_NoIO.S:Could not connect to Derby Network Server on host 127.0.0.1, port 1527: Connection refused
          at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun.ping(ReplicationRun.java:2733)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun.pingServer(ReplicationRun.java:2705)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun.startServer(ReplicationRun.java:2382)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun_Local.testReplication_Local_existingTestsAsReplLoad(ReplicationRun_Local.java:90)
          ------------------------

          During the running of the test, i.e. in the output above the failure report, it shows:
          testReplication_Local_existingTestsAsReplLoad <feff>java.io.IOException:
          java.io.IOException: ++++: not found
          at java.lang.UNIXProcess.<init>(Unknown Source)
          at java.lang.ProcessImpl.start(Unknown Source)
          at java.lang.ProcessBuilder.start(Unknown Source)
          at java.lang.Runtime.exec(Unknown Source)
          at java.lang.Runtime.exec(Unknown Source)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.
          ReplicationRun$4.run(ReplicationRun.java:2366)
          at java.lang.Thread.run(Unknown Source)
          java.lang.Exception: DRDA_NoIO.S:Could not connect to Derby Network Server on
          host 127.0.0.1, port 1527: Connection refused
          at org.apache.derby.impl.drda.NetworkServerControlImpl.
          consolePropertyMessageWork(NetworkServerControlImpl.java:3179)
          at org.apache.derby.impl.drda.NetworkServerControlImpl.
          consolePropertyMessage(NetworkServerControlImpl.java:1861)
          at org.apache.derby.impl.drda.NetworkServerControlImpl.setUpSocket
          (NetworkServerControlImpl.java:2502)
          at org.apache.derby.impl.drda.NetworkServerControlImpl.ping
          (NetworkServerControlImpl.java:1143)
          at org.apache.derby.drda.NetworkServerControl.ping(NetworkServerControl.
          java:395)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.
          ReplicationRun.ping(ReplicationRun.java:2718)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.
          ReplicationRun.pingServer(ReplicationRun.java:2705)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.
          ReplicationRun.startServer(ReplicationRun.java:2382)
          at org.apache.derbyTesting.functionTests.tests.replicationTests.
          ReplicationRun_Local.testReplication_Local_existingTestsAsReplLoad
          (ReplicationRun_Local.java:90)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at junit.framework.TestCase.runTest(TestCase.java:154)
          at junit.framework.TestCase.runBare(TestCase.java:127)
          at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:105)
          at junit.framework.TestResult$1.protect(TestResult.java:106)
          at junit.framework.TestResult.runProtected(TestResult.java:124)
          at junit.framework.TestResult.run(TestResult.java:109)
          at junit.framework.TestCase.run(TestCase.java:118)
          at junit.framework.TestSuite.runTest(TestSuite.java:208)
          at junit.framework.TestSuite.run(TestSuite.java:203)
          at junit.framework.TestSuite.runTest(TestSuite.java:208)
          at junit.framework.TestSuite.run(TestSuite.java:203)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.framework.TestResult.runProtected(TestResult.java:124)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at junit.textui.TestRunner.doRun(TestRunner.java:116)
          at junit.textui.TestRunner.start(TestRunner.java:172)
          at junit.textui.TestRunner.main(TestRunner.java:138)
          -------------------
          This output has been edited to not go past 80 chars, and the cut-and-paste actions shows the 'not found' characters as +++ while in the original it was more like "├╛├┐"

          Not sure if this is worth looking into, but I thought I'd report it here.

          Show
          Myrna van Lunteren added a comment - Manjula's test run (see DERBY-4019 ) suggests that the run is ok... However, I did some experiments with running the ReplicationRun_Local test with the property -Dfile.encoding="UTF-16", which I think only works with (sun) jdks 1.5 and up, and should also test encoding-safe usage. Running that test like that with jdk 1.5 passed consistenly on windows, but consistently failed on linux: ------------------------ 1) testReplication_Local_existingTestsAsReplLoad(org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun_Local)java.lang.Exception: Could not ping in 150 * 500ms.: DRDA_NoIO.S:Could not connect to Derby Network Server on host 127.0.0.1, port 1527: Connection refused at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun.ping(ReplicationRun.java:2733) at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun.pingServer(ReplicationRun.java:2705) at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun.startServer(ReplicationRun.java:2382) at org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationRun_Local.testReplication_Local_existingTestsAsReplLoad(ReplicationRun_Local.java:90) ------------------------ During the running of the test, i.e. in the output above the failure report, it shows: testReplication_Local_existingTestsAsReplLoad <feff>java.io.IOException: java.io.IOException: ++++: not found at java.lang.UNIXProcess.<init>(Unknown Source) at java.lang.ProcessImpl.start(Unknown Source) at java.lang.ProcessBuilder.start(Unknown Source) at java.lang.Runtime.exec(Unknown Source) at java.lang.Runtime.exec(Unknown Source) at org.apache.derbyTesting.functionTests.tests.replicationTests. ReplicationRun$4.run(ReplicationRun.java:2366) at java.lang.Thread.run(Unknown Source) java.lang.Exception: DRDA_NoIO.S:Could not connect to Derby Network Server on host 127.0.0.1, port 1527: Connection refused at org.apache.derby.impl.drda.NetworkServerControlImpl. consolePropertyMessageWork(NetworkServerControlImpl.java:3179) at org.apache.derby.impl.drda.NetworkServerControlImpl. consolePropertyMessage(NetworkServerControlImpl.java:1861) at org.apache.derby.impl.drda.NetworkServerControlImpl.setUpSocket (NetworkServerControlImpl.java:2502) at org.apache.derby.impl.drda.NetworkServerControlImpl.ping (NetworkServerControlImpl.java:1143) at org.apache.derby.drda.NetworkServerControl.ping(NetworkServerControl. java:395) at org.apache.derbyTesting.functionTests.tests.replicationTests. ReplicationRun.ping(ReplicationRun.java:2718) at org.apache.derbyTesting.functionTests.tests.replicationTests. ReplicationRun.pingServer(ReplicationRun.java:2705) at org.apache.derbyTesting.functionTests.tests.replicationTests. ReplicationRun.startServer(ReplicationRun.java:2382) at org.apache.derbyTesting.functionTests.tests.replicationTests. ReplicationRun_Local.testReplication_Local_existingTestsAsReplLoad (ReplicationRun_Local.java:90) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at junit.framework.TestCase.runTest(TestCase.java:154) at junit.framework.TestCase.runBare(TestCase.java:127) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:105) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.extensions.TestSetup.run(TestSetup.java:23) at junit.textui.TestRunner.doRun(TestRunner.java:116) at junit.textui.TestRunner.start(TestRunner.java:172) at junit.textui.TestRunner.main(TestRunner.java:138) ------------------- This output has been edited to not go past 80 chars, and the cut-and-paste actions shows the 'not found' characters as +++ while in the original it was more like "├╛├┐" Not sure if this is worth looking into, but I thought I'd report it here.
          Hide
          Myrna van Lunteren added a comment -

          I just noticed that org.apache.derbyTesting.functionTests.test.replicationTests.ReplicationRun, which is inherited by a number of the replication tests, uses InputStreamReader(InputStream). (e.g. processOutput).
          Not sure it's ok in this case and I can't verify this running ok with other encodings right now, but it should get checked into.

          Show
          Myrna van Lunteren added a comment - I just noticed that org.apache.derbyTesting.functionTests.test.replicationTests.ReplicationRun, which is inherited by a number of the replication tests, uses InputStreamReader(InputStream). (e.g. processOutput). Not sure it's ok in this case and I can't verify this running ok with other encodings right now, but it should get checked into.
          Hide
          Andrew McIntyre added a comment -

          Unsetting Fix Version for unassigned issues.

          Show
          Andrew McIntyre added a comment - Unsetting Fix Version for unassigned issues.
          Hide
          Rick Hillegas added a comment -

          Move to 10.2.3.0.

          Show
          Rick Hillegas added a comment - Move to 10.2.3.0.
          Hide
          Rick Hillegas added a comment -

          Moving to 10.2.2.0.

          Show
          Rick Hillegas added a comment - Moving to 10.2.2.0.
          Hide
          Daniel John Debrunner added a comment -

          The complete class StringBufferInputStream is also non-portable and deprecated. It's used in ClobOutputStream though fixing DERBY-684 might remove its use.

          Show
          Daniel John Debrunner added a comment - The complete class StringBufferInputStream is also non-portable and deprecated. It's used in ClobOutputStream though fixing DERBY-684 might remove its use.

            People

            • Assignee:
              Unassigned
              Reporter:
              Sunitha Kambhampati
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development