[VELOCITY-880] DataSourceResourceLoader corrupts UTF-8 encoded characters in template

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.x
• Fix Version/s: 2.0
• Environment: Oracle12c and HSQLDB 2.3.4, JDK 1.8
• Flags: Patch
• Assignee: Claude Brisson
• Reporter: James R Doyle
• Created: 16/May/17 21:59
• Updated: 07/Jan/18 15:46
• Resolved: 26/Jun/17 13:51

Description

A long-withstanding bug in the DataSourceResourceLoader corrupts UTF-8 templates retrieved from the database. The Unit Test suite for this resource loader has deficiencies that hide the bug.

The cause of the problem is this:

      InputStream rawStream = rs.getAsciiStream(templateColumn);

The resolution of the problem is simply:
      Reader r = rs.getCharacterStream(templateColumn);
      InputStream rawStream = null;
           try {
                rawStream = IOUtils.toInputStream(IOUtils.toString(r), encoding);
                } catch (IOException ioe) {}

Once done, the test failure vanishes:
org.junit.ComparisonFailure: Unicode test failed.
Expected :The Euro Currency Symbol € is a two-byte UTF-8 encoded character.
Actual :The Euro Currency Symbol ? is a two-byte UTF-8 encoded character.

The bug was verified and the fix was tested against Oracle12c and HSQLDB 2.3.4 using a CLOB column to store the template data.

The Unit Tests for this resource loader need attention.

Please see VELOCITY-599 ; long standing problem, which has been erroneously marked as resolved but has been in the codebase for a long time.

Issue Links

• relates to: VELOCITY-599 DataSourceResourceLoader doesn't support UTF8