NekoHTMLTransformer needs to set the default-encoding of the current system to work properly with UTF-8

• Key: COCOON-2063
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Minor
• Assignee: Jörg Heinicke
• Reporter: Alexander Klimetschek
• Component/s: Blocks: HTML
• Affects Version/s: 2.1.11, 2.2
• Fix Version/s: 2.1.12-dev (Current SVN), 2.2-dev (Current SVN)

Description

The NekoHTMLTransformer uses the cyberneko HTMLConfiguration for tidying html. Unfortunately it does not use the system's current encoding as default, instead you have to set a property to set your encoding. But this varies from one OS to another, so the best solution is to set this property automatically in the NekoHTMLTransformer depending on what Java uses as defaultCharset:

            config.setProperty("http://cyberneko.org/html/properties/default-encoding", Charset.defaultCharset().name());

Comments

Affects cocoon-html-impl.

[Alexander Klimetschek]

I forgot to mention that if someone wants to override this property via the configuration of the NekoHTMLTransformer, he can certainly do it. The manual config is applied after the dynamic setting of the encoding property, thus the manual one overrides the dynamic one.

[Ellis Pritchard]

This has bitten us too.

Here's a patch for Cocoon 2.1.X, rev 597695

[Ellis Pritchard]

Added Affects Version 2.1.11-dev

[Ellis Pritchard]

Anyone fancy applying the patches?

[Jörg Heinicke]

I fixed this issue in 2.2. The fix in 2.1 does not work since it uses java.nio which was only added in Java 1.4. Cocoon 2.1 has to be Java 1.3 compatible. Is there a way to find out the default encoding in Java 1.3? All the classes and methods were it would be necessary like new String(byte[]) or InputStreamReader() only point to http://java.sun.com/j2se/1.3/docs/api/java/lang/package-summary.html#charenc which just names some constants. With Mac OS X I also have no access to the source code of the JDK. The bytecode implies that the mentioned classes and methods use some Sun-internal class to retrieve the default encoding.

[Jörg Heinicke]

Also had to revert the fix for Cocoon 2.2 since Charset.defaultCharset() is only available on Java 5.

[Ellis Pritchard]

Oh, how annoying.

Is there a possibility of starting to use the src/jdk1.x directories for these kind of patches?

Just because some people are stuck in the dark ages doesn't mean we can't shine a light...

[Jörg Heinicke]

With Ant this would be rather easy as we already have some 1.4-specific code in Cocoon 2.1. No idea about Maven.

Is there really no other way to figure out the default encoding on before Java 5 JVMs?

[Vadim Gritsenko]

I'm not sure why generator needs to know encoding... Can it be simply always set to UTF-8?

[Jörg Heinicke]

As Vadim mentioned at http://marc.info/?l=xml-cocoon-dev&m=120905050708311&w=4 the NekoHTMLTransformer had an issue with converting String to byte[] using OS' default encoding rather than keeping the string - and so had the NekoHTMLGenerator when reading a request parameter value. These both issues are fixed in SVN and maybe caused the symptoms you saw. Closing the issue for now. Feel free to reopen it if problem still persists.