Details
-
Type:
Bug
-
Status: Open
-
Priority:
Critical
-
Resolution: Unresolved
-
Affects Version/s: 0.9, 0.10-dev
-
Fix Version/s: None
-
Component/s: Internationalisation (i18n)
-
Labels:None
Description
We're using Forrest to generate the Apache ManifoldCF site. We've added Japanese content. The content worked fine via localhost:8888, but the generated html content does not load properly in a browser, even though the browser correctly divines that the HTML page has utf-8 encoding. It looks like many utf-8 characters in the source XML are handled correctly but some are corrupted. I've also tried the fix in FORREST-668 but this does not help. See http://incubator.apache.org/connectors and click on the tab in Japanese to see what I mean. The current source for the site can be found in: https://svn.apache.org/repos/asf/incubator/lcf/trunk/site.
I checked out latest Forrest trunk and built and used that but there has been no improvement.
I checked out latest Forrest trunk and built and used that but there has been no improvement.
Issue Links
- blocks
-
CONNECTORS-385
Manifold CF site japanese is corrupted
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Hide
Hitoshi Ozawa
added a comment -
Permalink
While at this, would appreciate if it's possible to install Japanese fonts as well so pdf containing Japanese would show up correctly as well.
Show
Hitoshi Ozawa
added a comment - While at this, would appreciate if it's possible to install Japanese fonts as well so pdf containing Japanese would show up correctly as well.
Please ask about separate usage issues on the user mailing list.
The PDF fonts are configurable. See that plugin's docs:
http://forrest.apache.org/docs/plugins/org.apache.forrest.plugin.output.pdf/
The PDF fonts are configurable. See that plugin's docs:
http://forrest.apache.org/docs/plugins/org.apache.forrest.plugin.output.pdf/
I'm told that the Japanese portion of the site is correctly generated on a system that has a default locale of ja_JP. Obviously, though, this is not a good solution to the problem since we cannot select different locales when there is more than one language involved.
Sorry David, I thought the html pages were being dynamically generated on the Apache server.
It seems it's not. "forrest site" works fine on my Japanese OS.
Karl, is your system setup to use en_US-UTF-8?
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
It seems it's not. "forrest site" works fine on my Japanese OS.
Karl, is your system setup to use en_US-UTF-8?
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
bq. Karl, is your system setup to use en_US-UTF-8?
bq. export LC_ALL=en_US.UTF-8
bq. export LANG=en_US.UTF-8
bq. export LANGUAGE=en_US.UTF-8
I set the equivalent Windows variables but no change in the generated code for me. So it must be something else.
bq. export LC_ALL=en_US.UTF-8
bq. export LANG=en_US.UTF-8
bq. export LANGUAGE=en_US.UTF-8
I set the equivalent Windows variables but no change in the generated code for me. So it must be something else.
I figured it out. What we need to do is set the JAVA default encoding to UTF-8. The easy way to do this is (on Windows):
set JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8
... or on Linux:
export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8
Doing this before a Forrest invocation causes all JVMs it brings up to have the right encoding. (It's Cocoon that seems to be broken, by the way)
set JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8
... or on Linux:
export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8
Doing this before a Forrest invocation causes all JVMs it brings up to have the right encoding. (It's Cocoon that seems to be broken, by the way)
This patch works, at least as far as generating Japanese correctly on an en_US Windows machine.
Thanks. I was thinking of a similar patch. However i wondered if it would need to append this setting to any existing JAVA_TOOL_OPTIONS then reset at finish.
I have applied your patch as-is. Thanks.
If someone thinks that it needs more, then please do.
Regarding the Cocoon situation, i think that the doc comments refer to the fact that Cocoon/Forrest have many supporting products handling various parts of the system. Perhaps some of those treat the encoding differently. So this environment setting seems a good solution.
I have applied your patch as-is. Thanks.
If someone thinks that it needs more, then please do.
Regarding the Cocoon situation, i think that the doc comments refer to the fact that Cocoon/Forrest have many supporting products handling various parts of the system. Perhaps some of those treat the encoding differently. So this environment setting seems a good solution.