Derby
  1. Derby
  2. DERBY-4035

i18n tests fail with Lexical error on z/os

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.3.3.0, 10.5.1.1
    • Fix Version/s: 10.11.0.0, 10.10.2.0
    • Component/s: Test
    • Environment:
    • Urgency:
      Normal
    • Bug behavior facts:
      Regression Test Failure

      Description

      On z/os the following tests fail with a Lexical error e.g.
      ERROR 42X02: Lexical error at line 1, column 1. Encountered: "`" (96), after : "". There also seems to be some garbage in the output. The tests that fail with these errors are:
      derbyall/derbyall.fail:i18n/JapanCodeConversion.sql
      derbyall/derbyall.fail:i18n/UnicodeEscape_JP.sql
      derbyall/derbyall.fail:i18n/I18NImportExport.sql
      derbyall/derbyall.fail:i18n/urlLocale.sql
      derbyall/derbyall.fail:i18n/messageLocale.sql
      derbyall/derbyall.fail:i18n/caseI_tr_TR.sql
      derbyall/derbyall.fail:i18n/iepnegativetests_ES.sql

      The error I think is coming from ij, it is not in the derby.log and there is not a stack trace anywhere.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Kathey Marsden added a comment -

          This seems to be related to the fact that the i18n tests are run with derby.ui.codeset=UTF8. The input sql file is EBCDIC on Z so this really mangles things. The change to have the i18n tests run with derby.ui.codeset=UTF8 was made as the fix for DERBY-244.

          If I force the derby.ui.codeset to EBCDIC the test doesn't pass but fails only with a bit of garbage of the form EnC:>65533< EnC:>655533... at the beginning of the file before the version prints. The rest of the output is as expected.

          Show
          Kathey Marsden added a comment - This seems to be related to the fact that the i18n tests are run with derby.ui.codeset=UTF8. The input sql file is EBCDIC on Z so this really mangles things. The change to have the i18n tests run with derby.ui.codeset=UTF8 was made as the fix for DERBY-244 . If I force the derby.ui.codeset to EBCDIC the test doesn't pass but fails only with a bit of garbage of the form EnC:>65533< EnC:>655533... at the beginning of the file before the version prints. The rest of the output is as expected.
          Hide
          Knut Anders Hatlen added a comment -

          Kathey, I'm not sure I understand what you mean by "[the] input sql file is EBCDIC on Z". Are there separate input sql files for Z for these tests, and if so, where can I find them?

          Show
          Knut Anders Hatlen added a comment - Kathey, I'm not sure I understand what you mean by " [the] input sql file is EBCDIC on Z". Are there separate input sql files for Z for these tests, and if so, where can I find them?
          Hide
          Kathey Marsden added a comment -

          The test harness rewrites the sql file in the native encoding to the test directory and then runs the test from there. So after you run the test (if it fails or if you run with keepfiels) you see in the test directory the sql file, for example: JapanCodeConversion.sql

          It is this sql file that is in the native encoding and is causing the problem when fed to ij with derby.ui.codeset=UTF-8.

          In the RunTest code, this copy happens I think near line 710 where we have:
          710:
          // Read the test file and copy it to the outDir
          // except for multi tests (for multi we just need to locate it)
          BufferedReader in = new BufferedReader(new InputStreamReader(is, "UTF-8"));
          ...
          715
          script = new File((new File(userdir, scriptFileName)).getCanonicalPath());
          ...
          732:
          PrintWriter pw = null;
          pw = new PrintWriter( new BufferedWriter
          (new FileWriter(script.getPath()), 10000), true );

          String str = "";
          while ( (str = in.readLine()) != null )

          { pw.println(str); }
          Show
          Kathey Marsden added a comment - The test harness rewrites the sql file in the native encoding to the test directory and then runs the test from there. So after you run the test (if it fails or if you run with keepfiels) you see in the test directory the sql file, for example: JapanCodeConversion.sql It is this sql file that is in the native encoding and is causing the problem when fed to ij with derby.ui.codeset=UTF-8. In the RunTest code, this copy happens I think near line 710 where we have: 710: // Read the test file and copy it to the outDir // except for multi tests (for multi we just need to locate it) BufferedReader in = new BufferedReader(new InputStreamReader(is, "UTF-8")); ... 715 script = new File((new File(userdir, scriptFileName)).getCanonicalPath()); ... 732: PrintWriter pw = null; pw = new PrintWriter( new BufferedWriter (new FileWriter(script.getPath()), 10000), true ); String str = ""; while ( (str = in.readLine()) != null ) { pw.println(str); }
          Hide
          Knut Anders Hatlen added a comment -

          RunTest also has a section which sets file.encoding to UTF-8 when running the i18n tests. What happens if you change that too to EBCDIC? (We're not really supposed to change file.encoding, so that might explain why we have problems with these tests.) I'm not sure, but there may be some characters in the input files that are not possible to represent in EBCDIC so that we won't get these tests to run anyway as long as we copy the input files to the native encoding first (I don't remember, but I think that may have been the reason why we set file.encoding to UTF-8 for these tests, but apparently it didn't work as we had hoped).

          Now all the setting of different encoding flags in RunTest is getting very complex (derby.ui.codeset, console.encoding, file.encoding, derbyTesting.encoding all set to different values depending on platform, JVM, name of test), so I'm beginning to think that it is easier to convert these tests to JUnit and control the encoding per test, rather than trying to clean up the mess in RunTest.

          Show
          Knut Anders Hatlen added a comment - RunTest also has a section which sets file.encoding to UTF-8 when running the i18n tests. What happens if you change that too to EBCDIC? (We're not really supposed to change file.encoding, so that might explain why we have problems with these tests.) I'm not sure, but there may be some characters in the input files that are not possible to represent in EBCDIC so that we won't get these tests to run anyway as long as we copy the input files to the native encoding first (I don't remember, but I think that may have been the reason why we set file.encoding to UTF-8 for these tests, but apparently it didn't work as we had hoped). Now all the setting of different encoding flags in RunTest is getting very complex (derby.ui.codeset, console.encoding, file.encoding, derbyTesting.encoding all set to different values depending on platform, JVM, name of test), so I'm beginning to think that it is easier to convert these tests to JUnit and control the encoding per test, rather than trying to clean up the mess in RunTest.
          Hide
          Knut Anders Hatlen added a comment -

          Triaged for 10.5.2. Changing component to Test since the discussion indicates this is a problem with the old test harness.

          Show
          Knut Anders Hatlen added a comment - Triaged for 10.5.2. Changing component to Test since the discussion indicates this is a problem with the old test harness.
          Hide
          Kathey Marsden added a comment -

          Attaching patch (derby-4035_I18NImportExport_diff.txt) to bring i18n/I18NImportExport in as a ScriptTestCase. Not converting because ij functionality LOCALIZED DISPLAY ON is being tested.

          Show
          Kathey Marsden added a comment - Attaching patch (derby-4035_I18NImportExport_diff.txt) to bring i18n/I18NImportExport in as a ScriptTestCase. Not converting because ij functionality LOCALIZED DISPLAY ON is being tested.
          Hide
          Kathey Marsden added a comment -

          derby-4035_I18NImportExport_ieptestsnegative_diff.txt brings in the conversion also for ieptestnegativetests_ES.sql and removes the i18n derbyall suite.

          Running suites.All and derbyall.

          Show
          Kathey Marsden added a comment - derby-4035_I18NImportExport_ieptestsnegative_diff.txt brings in the conversion also for ieptestnegativetests_ES.sql and removes the i18n derbyall suite. Running suites.All and derbyall.
          Hide
          ASF subversion and git services added a comment -

          Commit 1492393 from Kathey Marsden
          [ https://svn.apache.org/r1492393 ]

          DERBY-4035 i18n tests fail with Lexical error on z/os

          Complete conversion of i18nTest suite.
          I18NImportExport change to ScriptTestCase
          iepnegativetests_ES changed to extend tools/ImportExportProcedureTest which has
          conversion for iepnegativetests

          Show
          ASF subversion and git services added a comment - Commit 1492393 from Kathey Marsden [ https://svn.apache.org/r1492393 ] DERBY-4035 i18n tests fail with Lexical error on z/os Complete conversion of i18nTest suite. I18NImportExport change to ScriptTestCase iepnegativetests_ES changed to extend tools/ImportExportProcedureTest which has conversion for iepnegativetests
          Hide
          ASF subversion and git services added a comment -

          Commit 1492652 from Kathey Marsden
          [ https://svn.apache.org/r1492652 ]

          DERBY-4035 i18n tests fail with Lexical error on z/os
          merge from trunk ...
          DERBY-6244
          convert i18n/caseI_tr_TR.sql to JUnit
          1489924

          DERBY-6246 convert i18n/urlLocale.sql to JUnit
          1490716
          1491364

          DERBY-6255 convert i18n/messageLocale.sql to JUnit
          1491542
          1491590
          1492088

          convert ieptestsnegative_ES and I18NImportExport
          1492393

          Show
          ASF subversion and git services added a comment - Commit 1492652 from Kathey Marsden [ https://svn.apache.org/r1492652 ] DERBY-4035 i18n tests fail with Lexical error on z/os merge from trunk ... DERBY-6244 convert i18n/caseI_tr_TR.sql to JUnit 1489924 DERBY-6246 convert i18n/urlLocale.sql to JUnit 1490716 1491364 DERBY-6255 convert i18n/messageLocale.sql to JUnit 1491542 1491590 1492088 convert ieptestsnegative_ES and I18NImportExport 1492393

            People

            • Assignee:
              Kathey Marsden
              Reporter:
              Kathey Marsden
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development