JSPWiki
  1. JSPWiki
  2. JSPWIKI-396

UTF-8 characters in wiki pages incorrectly rendered if served by Weblogic

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.9
    • Fix Version/s: 2.9.1
    • Component/s: None
    • Labels:
      None

      Description

      The Germain Main.txt starts with Herzlichen Glückwunsch.

      If the page is served by Weblogic Server, the umlaut is rendered with FFC3 and FFBC in Boxes, both with Firefox and IE. Served by Geronimo, it's fine.

      Herzlichen Glᅢᄐckwunsch

      Firefox page info says, page encoding is UTF-8.

      1. WikiJSPFilter.java.diff
        1 kB
        Jürgen Weber
      2. screenshot-1.jpg
        37 kB
        Jürgen Weber
      3. main_de.png
        61 kB
        Juan Pablo Santos Rodríguez
      4. GlenJSP396.patch
        2 kB
        Glen Mazza
      5. .jpg
        3 kB
        Jürgen Weber
      6. .jpg
        3 kB
        Jürgen Weber

        Issue Links

          Activity

          Hide
          Harry Metske added a comment -

          Fixed in 2.10.2-svn-1 (making the compare case-insensitive)

          Show
          Harry Metske added a comment - Fixed in 2.10.2-svn-1 (making the compare case-insensitive)
          Hide
          Jürgen Weber added a comment -

          I tried jspwiki-2.10.1-rc1 with WebSphere Application Server 8.5.5.2, there the problem resurfaces.

          Reason is that UtilJ2eeCompat.initCompatibilityOptions() tries to match Websphere wheres serverInfo is "IBM WebSphere Application Server/8.5"

          A fix is to use a capital S. A little better would be to lowercase serverInfo before comparing.

          A lot better would be to throw out the server check mess and make the code compatible to all servers.

          Show
          Jürgen Weber added a comment - I tried jspwiki-2.10.1-rc1 with WebSphere Application Server 8.5.5.2, there the problem resurfaces. Reason is that UtilJ2eeCompat.initCompatibilityOptions() tries to match Websphere wheres serverInfo is "IBM WebSphere Application Server/8.5" A fix is to use a capital S. A little better would be to lowercase serverInfo before comparing. A lot better would be to throw out the server check mess and make the code compatible to all servers.
          Hide
          Harry Metske added a comment -

          Considering it fixed.

          Show
          Harry Metske added a comment - Considering it fixed.
          Hide
          Harry Metske added a comment -

          Good to hear !
          Thanks for the feedback.
          regards,
          Harry

          Show
          Harry Metske added a comment - Good to hear ! Thanks for the feedback. regards, Harry
          Hide
          Adrien Beau added a comment -

          As others have reported, the encoding worked fine for me in version 2.9.0, and is broken in 2.9.1. I have built version 2.10.0-svn-10, and can confirm that it works again!

          My setup:

          • Windows XP SP3 32-bits
          • OS Locale: French
          • OS Encoding: Windows-1252
          • Oracle JRE 7u21
          • Tomcat 7.0.41
          Show
          Adrien Beau added a comment - As others have reported, the encoding worked fine for me in version 2.9.0, and is broken in 2.9.1. I have built version 2.10.0-svn-10, and can confirm that it works again! My setup: Windows XP SP3 32-bits OS Locale: French OS Encoding: Windows-1252 Oracle JRE 7u21 Tomcat 7.0.41
          Hide
          Harry Metske added a comment -

          It took me quite a few hours but I think I fixed it now (2.10.0-svn-2).
          I have tested with Tomcat only, but with all combinations of LANG envvar, property jspwiki.encoding and the encoding of the files.
          They all show properly now in the browser.
          Let me know how it works on other containers.

          Regards,
          Harry

          Show
          Harry Metske added a comment - It took me quite a few hours but I think I fixed it now (2.10.0-svn-2). I have tested with Tomcat only, but with all combinations of LANG envvar, property jspwiki.encoding and the encoding of the files. They all show properly now in the browser. Let me know how it works on other containers. Regards, Harry
          Hide
          Harry Metske added a comment -

          The erronous behavior still is (regardless container) with the following OOTB scenario:

          • set OS envvar to LANG=en_US.ISO-8859-1 (at least not UTF-8)
          • leave jspwiki.encoding=UTF-8 (the default)
          • sample pages (which are UTF-8) should be rendered properly, including special characters:

          € (Hex UTF-8) : E2 82 AC (Unicode code point U+20AC)
          € (Hex ISO-8859-1): 80
          é (Hex UTF-8) : C3 A9 (Unicode code point U+00E9)
          é (Hex ISO-8859-1): E9

          Show
          Harry Metske added a comment - The erronous behavior still is (regardless container) with the following OOTB scenario: set OS envvar to LANG=en_US.ISO-8859-1 (at least not UTF-8) leave jspwiki.encoding=UTF-8 (the default) sample pages (which are UTF-8) should be rendered properly, including special characters: € (Hex UTF-8) : E2 82 AC (Unicode code point U+20AC) € (Hex ISO-8859-1): 80 é (Hex UTF-8) : C3 A9 (Unicode code point U+00E9) é (Hex ISO-8859-1): E9
          Hide
          Jürgen Weber added a comment -

          Yes, I had file.encoding=ISO-8859-1

          After
          export LC_ALL=en_US.UTF-8
          export LANG=en_US.UTF-8
          export LANGUAGE=en_US.UTF-8

          the German Wikipage displays OK.

          But I do have jspwiki.encoding =UTF-8
          so the setting must be ignored someplace.

          Show
          Jürgen Weber added a comment - Yes, I had file.encoding=ISO-8859-1 After export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 export LANGUAGE=en_US.UTF-8 the German Wikipage displays OK. But I do have jspwiki.encoding =UTF-8 so the setting must be ignored someplace.
          Hide
          Harry Metske added a comment -

          Jürgen, I tried the suggested page-encoding, but it did not make any difference.
          The only thing that makes it work properly is the file.encoding=UTF-8 Java system property (which is the default on most modern Linuxes).
          So, my theory is that somewhere in JSPWiki (or a library we use) we implicitly use this system property, while in other places we use the encoding that is configured in jspwiki.properties (jspwiki.encoding = UTF-8).

          To confirm this theory, could you run this simple jsp and tell me what the output is :

          <%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
          <html>
          <body>
          <%
          out.println("file.encoding=" + System.getProperty("file.encoding"));
          %>
          </body>
          </html>
          
          Show
          Harry Metske added a comment - Jürgen, I tried the suggested page-encoding, but it did not make any difference. The only thing that makes it work properly is the file.encoding=UTF-8 Java system property (which is the default on most modern Linuxes). So, my theory is that somewhere in JSPWiki (or a library we use) we implicitly use this system property, while in other places we use the encoding that is configured in jspwiki.properties (jspwiki.encoding = UTF-8). To confirm this theory, could you run this simple jsp and tell me what the output is : <%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%> <html> <body> <% out.println("file.encoding=" + System.getProperty("file.encoding")); %> </body> </html>
          Hide
          Jürgen Weber added a comment -

          No, I followed the "Really simple installation" from
          JSPWiki-2.9.1-incubating-4-bin.zip\JSPWiki-bin\README

          I think the Wiki should run out of the box without having to mess with container setup. Maybe the <page-encoding> entry in web.xml would work ..
          http://stackoverflow.com/questions/11400989/character-encoding-problems-with-tomcat

          Show
          Jürgen Weber added a comment - No, I followed the "Really simple installation" from JSPWiki-2.9.1-incubating-4-bin.zip\JSPWiki-bin\README I think the Wiki should run out of the box without having to mess with container setup. Maybe the <page-encoding> entry in web.xml would work .. http://stackoverflow.com/questions/11400989/character-encoding-problems-with-tomcat
          Hide
          Harry Metske added a comment -

          I cannot reproduce this problem.
          Are you sure you are running Tomcat in UTF-8 mode ( -Dfile.encoding=UTF-8 ) ?

          Show
          Harry Metske added a comment - I cannot reproduce this problem. Are you sure you are running Tomcat in UTF-8 mode ( -Dfile.encoding=UTF-8 ) ?
          Hide
          Jürgen Weber added a comment -

          Bug appears now with tomcat-7.0.39 and 2.9.1-incubating-4-rc2 and

          Show
          Jürgen Weber added a comment - Bug appears now with tomcat-7.0.39 and 2.9.1-incubating-4-rc2 and
          Hide
          Harry Metske added a comment -

          fixed in 2.9.1-incubating-1

          Show
          Harry Metske added a comment - fixed in 2.9.1-incubating-1
          Hide
          Marco Roeland added a comment -

          Simply adding "useStream=true" in src/org/apache/wiki/util/UtilJ2eeCompat.java for Tomcat fixes the issue for me. Perhaps it is time to reverse the default, as more and more implementations seem to require this, perhaps triggered by stricter or better UTF-8 support? For testing/debugging/workarounds the useStream setting might be made configurable in jspwiki.properties?

          Show
          Marco Roeland added a comment - Simply adding "useStream=true" in src/org/apache/wiki/util/UtilJ2eeCompat.java for Tomcat fixes the issue for me. Perhaps it is time to reverse the default, as more and more implementations seem to require this, perhaps triggered by stricter or better UTF-8 support? For testing/debugging/workarounds the useStream setting might be made configurable in jspwiki.properties?
          Hide
          Harry Metske added a comment -

          I will further investigate (if I can find some time the next few days).

          Show
          Harry Metske added a comment - I will further investigate (if I can find some time the next few days).
          Hide
          Marco Roeland added a comment -

          Since 2.9.1-svn-25 (12/Feb/2013, "JSPWIKI-759 Resin 4 compatibility", subversion revision 1445289) I now see the same issue also with Tomcat 7.0.39 (with java 1.7.0_17). Versions before that work as expected with UTF-8, but starting with this release pages with UTF-8 "characters" are rendered sort of doubled. A lowercase 'i' with umlaut, UTF-8 encoded in hexadecimal as "c3 af" gets rendered as "ef bf 83 ef be af". The content-type is passed correctly and explicitly encoding 'URIEncoding="UTF-8"' in server.xml doesn't make any difference to that.

          Show
          Marco Roeland added a comment - Since 2.9.1-svn-25 (12/Feb/2013, " JSPWIKI-759 Resin 4 compatibility", subversion revision 1445289) I now see the same issue also with Tomcat 7.0.39 (with java 1.7.0_17). Versions before that work as expected with UTF-8, but starting with this release pages with UTF-8 "characters" are rendered sort of doubled. A lowercase 'i' with umlaut, UTF-8 encoded in hexadecimal as "c3 af" gets rendered as "ef bf 83 ef be af". The content-type is passed correctly and explicitly encoding 'URIEncoding="UTF-8"' in server.xml doesn't make any difference to that.
          Hide
          Glen Mazza added a comment -

          Thanks, Juergen (and for Janne for proposing the simple solution), I went with the useStream=true solution as it still results in the Selenium tests passing. I'm unsure whether WikiJSPFilter can indeed be written so it will work on all app servers (it would take me a while before I could possibly get to this), but we'll have your patch as a starting point if we go that route.

          Show
          Glen Mazza added a comment - Thanks, Juergen (and for Janne for proposing the simple solution), I went with the useStream=true solution as it still results in the Selenium tests passing. I'm unsure whether WikiJSPFilter can indeed be written so it will work on all app servers (it would take me a while before I could possibly get to this), but we'll have your patch as a starting point if we go that route.
          Hide
          Glen Mazza added a comment -

          Turned on streaming for WebLogic per Juergen's confirmation that this method works.

          Show
          Glen Mazza added a comment - Turned on streaming for WebLogic per Juergen's confirmation that this method works.
          Hide
          Jürgen Weber added a comment -

          Yes, useStream = true does the trick, too.

          I tried useStream = true for Tomcat, too, but this breaks umlauts in Tomcat.
          I think WikiJSPFilter should be rewritten in order to make it work on all JEE servers, without having to use server specific hacks..

          Show
          Jürgen Weber added a comment - Yes, useStream = true does the trick, too. I tried useStream = true for Tomcat, too, but this breaks umlauts in Tomcat. I think WikiJSPFilter should be rewritten in order to make it work on all JEE servers, without having to use server specific hacks..
          Hide
          Glen Mazza added a comment -

          Juergen, this is the file that Janne is talking about (and he comes from a land with lots of umlauts too, so he might know something about this issue):
          http://svn.apache.org/viewvc/incubator/jspwiki/trunk/src/org/apache/wiki/util/UtilJ2eeCompat.java?revision=1395798&view=markup

          Would you please try with Weblogic, that if you add "useStream = true;" just after line 131 to the file above, if that fixes your problem without need for your patch? If that's the case, fantastic, we can quickly add that line in without it affecting our Tomcat-run Selenium tests. Otherwise we'll need to figure out why the View Page Source tests fail with your patch. Thanks!

          Show
          Glen Mazza added a comment - Juergen, this is the file that Janne is talking about (and he comes from a land with lots of umlauts too, so he might know something about this issue): http://svn.apache.org/viewvc/incubator/jspwiki/trunk/src/org/apache/wiki/util/UtilJ2eeCompat.java?revision=1395798&view=markup Would you please try with Weblogic, that if you add "useStream = true;" just after line 131 to the file above, if that fixes your problem without need for your patch? If that's the case, fantastic, we can quickly add that line in without it affecting our Tomcat-run Selenium tests. Otherwise we'll need to figure out why the View Page Source tests fail with your patch. Thanks!
          Hide
          Janne Jalkanen added a comment -

          Actually, WikiJSPFilter is way more complicated than a simple cache. When a plugin needs to insert a resource (e.g. CSS) to the HEAD of the document, it notifies TemplateManager, who stores it. Then, WikiJSPFilter buffers the entire response to memory, emits the resource requests (CSS) to the right location in the respnse, and then flushes to the real output stream.

          So disabling WikiJSPFilter disables quite a few plugins.

          I think a better way is to play with UtilJ2eeCompat and see if setting the useStream to true would help. That should wrap the response in a byte[], not char[] or something to that effect.

          Show
          Janne Jalkanen added a comment - Actually, WikiJSPFilter is way more complicated than a simple cache. When a plugin needs to insert a resource (e.g. CSS) to the HEAD of the document, it notifies TemplateManager, who stores it. Then, WikiJSPFilter buffers the entire response to memory, emits the resource requests (CSS) to the right location in the respnse, and then flushes to the real output stream. So disabling WikiJSPFilter disables quite a few plugins. I think a better way is to play with UtilJ2eeCompat and see if setting the useStream to true would help. That should wrap the response in a byte[], not char[] or something to that effect.
          Hide
          Glen Mazza added a comment -

          Hi Juergen, the patch seems to work (at least doesn't break anything) with Tomcat – however, when I run our Selenium web tests (done via "ant clean webtests" from the jspwiki source directory) The "View Page Source" test located at the very end of all five Selenium web test suites fail (green (good) before I made the change but dark pink (bad) afterwards). You can see the results just by opening up the 5 HTML pages located in jspwiki/tests/reports/selenium after activating the webtests. (Note we have a "Rename Profile" that also fails in some test suites-both before and after patch-I've been meaning to look at but that's unrelated to this issue.)

          So I didn't yet commit the change to production yet. Would you be able to look at why this webtest fails with this patch – I guess the problem is that toString() is no longer working, but I may be wrong here. Perhaps someone else on the JSPWiki can also chime in here on the potential problem. Else, I may be able to look at this further myself later.

          Also, please use diff -u > file.patch (the -u option makes it easier to read and incorporate) for any subsequent patches.

          Show
          Glen Mazza added a comment - Hi Juergen, the patch seems to work (at least doesn't break anything) with Tomcat – however, when I run our Selenium web tests (done via "ant clean webtests" from the jspwiki source directory) The "View Page Source" test located at the very end of all five Selenium web test suites fail (green (good) before I made the change but dark pink (bad) afterwards). You can see the results just by opening up the 5 HTML pages located in jspwiki/tests/reports/selenium after activating the webtests. (Note we have a "Rename Profile" that also fails in some test suites- both before and after patch -I've been meaning to look at but that's unrelated to this issue.) So I didn't yet commit the change to production yet. Would you be able to look at why this webtest fails with this patch – I guess the problem is that toString() is no longer working, but I may be wrong here. Perhaps someone else on the JSPWiki can also chime in here on the potential problem. Else, I may be able to look at this further myself later. Also, please use diff -u > file.patch (the -u option makes it easier to read and incorporate) for any subsequent patches.
          Hide
          Glen Mazza added a comment -

          Patch I used to make change (should be same as Juergen's but I used diff -u instead).

          Show
          Glen Mazza added a comment - Patch I used to make change (should be same as Juergen's but I used diff -u instead).
          Hide
          Glen Mazza added a comment -

          Excellent Juergen, thanks for sticking with us 4+ years until we could get this done, even if it ended up you being the one supplying the patch... :/ It's great that we'll be able to get foreign languages running fine with WebLogic.

          I'll try to get to this on Sunday unless someone else more knowledgable has time to take this over. We do not have to test this on WebLogic, as you're already vouching it works. I guess the only thing for us is to confirm the change works with Tomcat – I'll try it with the English, German, and (for multi-byte char sets) Chinese resource files to confirm Tomcat is still happy with this change. Does the team know anything else I'd have to check? Also, team, if you see any problems offhand with Juergen's patch please let me know.

          Show
          Glen Mazza added a comment - Excellent Juergen, thanks for sticking with us 4+ years until we could get this done, even if it ended up you being the one supplying the patch... :/ It's great that we'll be able to get foreign languages running fine with WebLogic. I'll try to get to this on Sunday unless someone else more knowledgable has time to take this over. We do not have to test this on WebLogic, as you're already vouching it works. I guess the only thing for us is to confirm the change works with Tomcat – I'll try it with the English, German, and (for multi-byte char sets) Chinese resource files to confirm Tomcat is still happy with this change. Does the team know anything else I'd have to check? Also, team, if you see any problems offhand with Juergen's patch please let me know.
          Hide
          Jürgen Weber added a comment -

          OK, this is quite subtle. Weblogic relies on that the wrapping ServletOutputStream is an OutputStream, i.e. backed by a byte[]. For u umlaut (Unicode C3 BC) Weblogic strangely sends
          -61 FFFFFFFFFFFFFFC3
          -68 FFFFFFFFFFFFFFBC
          In byte[] this ends as C3 BC, new String(C3 BC) correctly gives u umlaut.
          But currently WikiJSPFilter uses a CharArrayWriter, which uses char[] and stores u umlaut as FFC3, FFBC.char[].toString() yields the strange characters one sees. For Tomcat this seems to work by chance, probably the ServletFilter returns C3 BC as two chars which the browser combines into the unicode char.

          U+00FC ü c3 bc u umlaut

          Actually, I don't see the need for WikiJSPFilter, it just caches, and caching should be left to the appserver.

          Anyway, the appended patch fixes the Weblogic problem and does not break Tomcat.

          Show
          Jürgen Weber added a comment - OK, this is quite subtle. Weblogic relies on that the wrapping ServletOutputStream is an OutputStream, i.e. backed by a byte[]. For u umlaut (Unicode C3 BC) Weblogic strangely sends -61 FFFFFFFFFFFFFFC3 -68 FFFFFFFFFFFFFFBC In byte[] this ends as C3 BC, new String(C3 BC) correctly gives u umlaut. But currently WikiJSPFilter uses a CharArrayWriter, which uses char[] and stores u umlaut as FFC3, FFBC.char[].toString() yields the strange characters one sees. For Tomcat this seems to work by chance, probably the ServletFilter returns C3 BC as two chars which the browser combines into the unicode char. U+00FC ü c3 bc u umlaut Actually, I don't see the need for WikiJSPFilter, it just caches, and caching should be left to the appserver. Anyway, the appended patch fixes the Weblogic problem and does not break Tomcat.
          Hide
          Jürgen Weber added a comment -

          WebLogic Server Version: 12.1.1.0
          Played a lot with weblogic.xml, no change.
          I added a jsp @page header to German Main.txt and added jsp extension, then loaded directly Main.txt.jsp, same problem. Copied Main.txt.jsp into another context, problem was gone, umlauts displayed correctly.
          Back to JSPWiki context, in web.xml commented out filter-mapping WikiJSPFilter, problem was gone, umlauts displayed correctly. Activated again WikiJSPFilter, umlauts wrong again.

          Show
          Jürgen Weber added a comment - WebLogic Server Version: 12.1.1.0 Played a lot with weblogic.xml, no change. I added a jsp @page header to German Main.txt and added jsp extension, then loaded directly Main.txt.jsp, same problem. Copied Main.txt.jsp into another context, problem was gone, umlauts displayed correctly. Back to JSPWiki context, in web.xml commented out filter-mapping WikiJSPFilter, problem was gone, umlauts displayed correctly. Activated again WikiJSPFilter, umlauts wrong again.
          Hide
          Glen Mazza added a comment -

          Hmm, Apache Roller had a similar problem recently with Java resource files: https://issues.apache.org/jira/browse/ROL-1955 that was fixed in this manner:
          http://svn.apache.org/viewvc/roller/trunk/weblogger-web/src/main/resources/ApplicationResources_de.properties?r1=895563&r2=1416323&diff_format=h

          I notice that our German property files do things in what Roller had determined to be the "bad" way:
          http://svn.apache.org/viewvc/incubator/jspwiki/trunk/etc/i18n/CoreResources_de.properties?revision=1400875&view=markup#l57

          We may need to switch to the Roller way of encoding special German characters. (It's what we do for Chinese already: http://svn.apache.org/viewvc/incubator/jspwiki/trunk/etc/i18n/CoreResources_zh_CN.properties?revision=1400875&view=markup ).

          However, our Main.text (and all other sample .text files) is where this problem is occurring, and I'm just seeing properly formatted umlaut characters in the .text files:
          http://svn.apache.org/viewvc/incubator/jspwiki/trunk/src/wikipages/de/Main.txt?revision=1234977&view=markup
          ...as well as actual Chinese characters (none of the "\u..." escaping stuff) in the Chinese files:
          http://svn.apache.org/viewvc/incubator/jspwiki/trunk/src/wikipages/zh_CN/Main.txt?revision=1234977&view=markup

          But this could be an issue with my browser and/or how SVN decides to render .text vs .properties files. Anyone know if we can use \uxxxx escaping in our Text files, and if it will work? I can test it... Only problem I see is, even if that does work and fixes this issue, what about German users typing in accented characters, those would need to be escaped too or there would be the same problem on WebLogic.

          Show
          Glen Mazza added a comment - Hmm, Apache Roller had a similar problem recently with Java resource files: https://issues.apache.org/jira/browse/ROL-1955 that was fixed in this manner: http://svn.apache.org/viewvc/roller/trunk/weblogger-web/src/main/resources/ApplicationResources_de.properties?r1=895563&r2=1416323&diff_format=h I notice that our German property files do things in what Roller had determined to be the "bad" way: http://svn.apache.org/viewvc/incubator/jspwiki/trunk/etc/i18n/CoreResources_de.properties?revision=1400875&view=markup#l57 We may need to switch to the Roller way of encoding special German characters. (It's what we do for Chinese already: http://svn.apache.org/viewvc/incubator/jspwiki/trunk/etc/i18n/CoreResources_zh_CN.properties?revision=1400875&view=markup ). However, our Main.text (and all other sample .text files) is where this problem is occurring, and I'm just seeing properly formatted umlaut characters in the .text files: http://svn.apache.org/viewvc/incubator/jspwiki/trunk/src/wikipages/de/Main.txt?revision=1234977&view=markup ...as well as actual Chinese characters (none of the "\u..." escaping stuff) in the Chinese files: http://svn.apache.org/viewvc/incubator/jspwiki/trunk/src/wikipages/zh_CN/Main.txt?revision=1234977&view=markup But this could be an issue with my browser and/or how SVN decides to render .text vs .properties files. Anyone know if we can use \uxxxx escaping in our Text files, and if it will work? I can test it... Only problem I see is, even if that does work and fixes this issue, what about German users typing in accented characters, those would need to be escaped too or there would be the same problem on WebLogic.
          Hide
          Jürgen Weber added a comment -

          Oracle Weblogic Server can be downloaded at
          http://www.oracle.com/technetwork/middleware/downloads/index.html
          Oracle gives a free license "for the purpose of developing, testing, prototyping and demonstrating your application".

          Show
          Jürgen Weber added a comment - Oracle Weblogic Server can be downloaded at http://www.oracle.com/technetwork/middleware/downloads/index.html Oracle gives a free license "for the purpose of developing, testing, prototyping and demonstrating your application".
          Hide
          Glen Mazza added a comment -

          Harry quote: "So Tomcat works fine, Geronimo does, Glassfish does, but WebLogic doesn't, what could we possibly fix in JSPWiki?"

          I agree that if it can run in multiple containers, the problem is with WebLogic then and not us--but I believe Glassfish and Geronimo both embed Tomcat so it may be an issue that we're incorrectly relying on Tomcat's behavior for something but not coding in a generic manner that will work on all servlet containers. Then again, WebLogic is not open source so we can't be expected to have local copies of it for our own testing.

          There may be nothing we can do about this, and I understand it is irritating keeping it open for that reason, but doing so allows us to immediately close similar reports as a duplicate of this one, as well as have other people accumulate responses about how they were able to get WebLogic working with JSPWiki. Ultimately, problems with major commercial servers like WebLogic or Websphere we can't be expected to test as we don't have the licenses, but we can keep the issue open, hoping users of those containers will see this matter and provide their solutions. For all we need is a commentor providing a solution and vouching that it works, and if there's no response from the original reporter that he has tested it to confirm it doesn't work, then this matter can be closed.

          Nobody likes closing dead issues more than me (in two weeks, I closed about 110 of Roller's, getting open issues down to 300) but this particular issue, because it involves such a major commercial server, seems to be something worth keeping open.

          Show
          Glen Mazza added a comment - Harry quote: "So Tomcat works fine, Geronimo does, Glassfish does, but WebLogic doesn't, what could we possibly fix in JSPWiki?" I agree that if it can run in multiple containers, the problem is with WebLogic then and not us--but I believe Glassfish and Geronimo both embed Tomcat so it may be an issue that we're incorrectly relying on Tomcat's behavior for something but not coding in a generic manner that will work on all servlet containers. Then again, WebLogic is not open source so we can't be expected to have local copies of it for our own testing. There may be nothing we can do about this, and I understand it is irritating keeping it open for that reason, but doing so allows us to immediately close similar reports as a duplicate of this one, as well as have other people accumulate responses about how they were able to get WebLogic working with JSPWiki. Ultimately, problems with major commercial servers like WebLogic or Websphere we can't be expected to test as we don't have the licenses, but we can keep the issue open, hoping users of those containers will see this matter and provide their solutions. For all we need is a commentor providing a solution and vouching that it works, and if there's no response from the original reporter that he has tested it to confirm it doesn't work, then this matter can be closed. Nobody likes closing dead issues more than me (in two weeks, I closed about 110 of Roller's, getting open issues down to 300) but this particular issue, because it involves such a major commercial server, seems to be something worth keeping open.
          Hide
          Glen Mazza added a comment -

          Can the user try these instructions: http://vhmdc.blogspot.com/2012/11/running-jspwiki-on-weblogic-11g.html? Maybe it might work then.

          Show
          Glen Mazza added a comment - Can the user try these instructions: http://vhmdc.blogspot.com/2012/11/running-jspwiki-on-weblogic-11g.html? Maybe it might work then.
          Hide
          Harry Metske added a comment -

          propose to close again...

          Show
          Harry Metske added a comment - propose to close again...
          Hide
          Florian Holeczek added a comment -

          Added 2.9 to the affected versions, though probably all previous ones are affected, too.

          However, I wouldn't consider this to be fixed in 2.9, since priority is to push out a graduation release.

          Show
          Florian Holeczek added a comment - Added 2.9 to the affected versions, though probably all previous ones are affected, too. However, I wouldn't consider this to be fixed in 2.9, since priority is to push out a graduation release.
          Hide
          Juan Pablo Santos Rodríguez added a comment -

          Tried all the solutions noted on those links, set weblogic to start with -Dfile.encoding=utf8 but nothing happens. The page contents are correctly saved but, if I create a page called "Ñaña", the page is saved as %C3%91a%C3%B1a.txt

          Also note that all characters are incorrectly rendered, not only the page contents (see attached screenshot).

          Seems to me that weblogic, somewhere, is UTF-encoding the contents twice. I'll get a look on this after I'm done with the maven-ant-tasks..

          Show
          Juan Pablo Santos Rodríguez added a comment - Tried all the solutions noted on those links, set weblogic to start with -Dfile.encoding=utf8 but nothing happens. The page contents are correctly saved but, if I create a page called "Ñaña", the page is saved as %C3%91a%C3%B1a.txt Also note that all characters are incorrectly rendered, not only the page contents (see attached screenshot). Seems to me that weblogic, somewhere, is UTF-encoding the contents twice. I'll get a look on this after I'm done with the maven-ant-tasks..
          Hide
          Florian Holeczek added a comment -

          Hi all,

          other OS projects had the same problem and some fixed it. See for example:

          http://jira.magnolia-cms.com/browse/MAGNOLIA-2087
          http://mail-archives.apache.org/mod_mbox/portals-jetspeed-user/200811.mbox/%3C1271b8570811240703i7f9ca70cg9eb1bfcafee523fd@mail.gmail.com%3E

          I didn't yet have a look at it in detail, but the solution seems to be very simple. If it doesn't affect compatibility with other environments, I propose to fix it in 2.9.1.

          Regards
          Florian

          Show
          Florian Holeczek added a comment - Hi all, other OS projects had the same problem and some fixed it. See for example: http://jira.magnolia-cms.com/browse/MAGNOLIA-2087 http://mail-archives.apache.org/mod_mbox/portals-jetspeed-user/200811.mbox/%3C1271b8570811240703i7f9ca70cg9eb1bfcafee523fd@mail.gmail.com%3E I didn't yet have a look at it in detail, but the solution seems to be very simple. If it doesn't affect compatibility with other environments, I propose to fix it in 2.9.1. Regards Florian
          Hide
          Harry Metske added a comment -

          propose to close again...

          Show
          Harry Metske added a comment - propose to close again...
          Hide
          Harry Metske added a comment -

          Jürgen,

          I can't think of anything going wrong in JSPWiki here, if you have set the property jspwiki.encoding=UTF-8.

          The hexdumped html page should show x'c3' x'bc' for an ü character, and the same for the wiki page on the file system.
          The http response header you should get is "Content-Type:text/html;charset=UTF-8".
          Then everything should work fine.

          I cannot reproduce the issue here, tried different browsers (with Tomcat).
          So Tomcat works fine, Geronimo does, Glassfish does, but WebLogic doesn't, what could we possibly fix in JSPWiki ?

          Show
          Harry Metske added a comment - Jürgen, I can't think of anything going wrong in JSPWiki here, if you have set the property jspwiki.encoding=UTF-8. The hexdumped html page should show x'c3' x'bc' for an ü character, and the same for the wiki page on the file system. The http response header you should get is "Content-Type:text/html;charset=UTF-8". Then everything should work fine. I cannot reproduce the issue here, tried different browsers (with Tomcat). So Tomcat works fine, Geronimo does, Glassfish does, but WebLogic doesn't, what could we possibly fix in JSPWiki ?
          Hide
          Jürgen Weber added a comment -

          Checked with JSPWiki v2.9.0-incubating-1 and WebLogic Server Version: 12.1.1.0, problem still there.
          A the downloaded German main page hexdumps as
          21a0 22 3e 48 65 72 7a 6c 69 63 68 65 6e 20 47 6c ef ">Herzlichen Gl.
          21b0 bf 83 ef be bc 63 6b 77 75 6e 73 63 68 21 3c 61 ....ckwunsch!<a

          Adding <charset-params> did not help.
          http://docs.oracle.com/cd/E15051_01/wls/docs103/webapp/weblogic_xml.html

          Show
          Jürgen Weber added a comment - Checked with JSPWiki v2.9.0-incubating-1 and WebLogic Server Version: 12.1.1.0, problem still there. A the downloaded German main page hexdumps as 21a0 22 3e 48 65 72 7a 6c 69 63 68 65 6e 20 47 6c ef ">Herzlichen Gl. 21b0 bf 83 ef be bc 63 6b 77 75 6e 73 63 68 21 3c 61 ....ckwunsch!<a Adding <charset-params> did not help. http://docs.oracle.com/cd/E15051_01/wls/docs103/webapp/weblogic_xml.html
          Hide
          Harry Metske added a comment -

          no follow-up
          re-open if any new insights pop-up

          Show
          Harry Metske added a comment - no follow-up re-open if any new insights pop-up
          Hide
          Harry Metske added a comment -

          Jürgen, Murray, any progress on this one ?

          Show
          Harry Metske added a comment - Jürgen, Murray, any progress on this one ?
          Hide
          Jürgen Weber added a comment -

          Murray, have your boxes different settings of the LANG environment variable? I suspect it now for being a cause.

          Show
          Jürgen Weber added a comment - Murray, have your boxes different settings of the LANG environment variable? I suspect it now for being a cause.
          Hide
          Murray Altheim added a comment -

          I spoke way too soon. It turns out that we still have the character encoding issue. We've narrowed the problem though, as we have identically-configured wikis, one on Ubuntu linux (where things work okay) and one on Solaris 10 (where we see the character encoding issue).

          The reason I thought it was fixed is that URL handling does work okay on Solaris, but entering diacritical vowels (from FF on Windows) causes problems only on Solaris, regardless of browser setting (i.e., even if the browser is set to UTF-8).

          I'm not sure if this is the same bug as JSPWIKI-396. still investigating, hopefully today.

          Show
          Murray Altheim added a comment - I spoke way too soon. It turns out that we still have the character encoding issue. We've narrowed the problem though, as we have identically-configured wikis, one on Ubuntu linux (where things work okay) and one on Solaris 10 (where we see the character encoding issue). The reason I thought it was fixed is that URL handling does work okay on Solaris, but entering diacritical vowels (from FF on Windows) causes problems only on Solaris, regardless of browser setting (i.e., even if the browser is set to UTF-8). I'm not sure if this is the same bug as JSPWIKI-396 . still investigating, hopefully today.
          Hide
          Harry Metske added a comment -

          Jürgen, we are talking two different things here

          1. page encoding, this is done by calling ServletRequest|ServletResponse.setCharacterEncoding() or if you use JSP:
          <%@ page pageEncoding="UTF-8" %> (see http://java.sun.com/j2ee/1.4/docs/api/index.html for more details), this results in a Content-Type http header specifying the UTF-8 codepage.

          2. setting the right character encoding om the tomcat connector in tomcat's server.xml , or an equivalent in WebLogic.

          Both have to be set properly, the first one is done by JSPWiki, and I think it does it correctly.

          Looking at your wget hexdump, that is really strange, you have twice a u with umlaut, both encoded differently and both wrong.
          I just did the same for tomcat, and that gives me a hex "c3 bc", which is correct according to http://www.utf8-chartable.de/. see attachment.

          It sure looks like something wrong with WebLogic to me too.

          Show
          Harry Metske added a comment - Jürgen, we are talking two different things here 1. page encoding, this is done by calling ServletRequest|ServletResponse.setCharacterEncoding() or if you use JSP: <%@ page pageEncoding="UTF-8" %> (see http://java.sun.com/j2ee/1.4/docs/api/index.html for more details), this results in a Content-Type http header specifying the UTF-8 codepage. 2. setting the right character encoding om the tomcat connector in tomcat's server.xml , or an equivalent in WebLogic. Both have to be set properly, the first one is done by JSPWiki, and I think it does it correctly. Looking at your wget hexdump, that is really strange, you have twice a u with umlaut, both encoded differently and both wrong. I just did the same for tomcat, and that gives me a hex "c3 bc", which is correct according to http://www.utf8-chartable.de/ . see attachment. It sure looks like something wrong with WebLogic to me too.
          Hide
          Janne Jalkanen added a comment -

          No idea what is going on then. It's as if Weblogic randomly decrements first byte of a valid UTF-8 character. If that is true, then Weblogic is badly broken and needs to be fixed.

          If you need just German, you should probably switch JSPWiki to Latin1 (with jspwiki.encoding=ISO-8859-1).

          Show
          Janne Jalkanen added a comment - No idea what is going on then. It's as if Weblogic randomly decrements first byte of a valid UTF-8 character. If that is true, then Weblogic is badly broken and needs to be fixed. If you need just German, you should probably switch JSPWiki to Latin1 (with jspwiki.encoding=ISO-8859-1).
          Hide
          Jürgen Weber added a comment -

          No, the encoding parameter did not change things. Actually, it seems to be deprecated and have moved into web.xml:

          http://forums.sun.com/thread.jspa?threadID=5195636&messageID=9771509
          http://www.oracle.com/technology/sample_code/tech/java/codesnippet/jsps/jspconfig/JSPConfig.html

          But still no change, umlauts wrong and Firefox still displaying UTF-8 as page encoding.

          I wgot the page and looked at it with a hex editor: see attachment

          When you look at the generated html source, the encoding comes very late in the page, might that be a cause?

          Show
          Jürgen Weber added a comment - No, the encoding parameter did not change things. Actually, it seems to be deprecated and have moved into web.xml: http://forums.sun.com/thread.jspa?threadID=5195636&messageID=9771509 http://www.oracle.com/technology/sample_code/tech/java/codesnippet/jsps/jspconfig/JSPConfig.html But still no change, umlauts wrong and Firefox still displaying UTF-8 as page encoding. I wgot the page and looked at it with a hex editor: see attachment When you look at the generated html source, the encoding comes very late in the page, might that be a cause?
          Hide
          Murray Altheim added a comment -

          Okay, can't speak for the weblogic issue, but for Apache/Tomcat 5.5/6 as per recommendation

          http://www.jspwiki.org/wiki/TomcatAndUTF8

          I added the attribute to server.xml as recommended, rebooted tomcat and things are working
          as expected. Thanks!

          Show
          Murray Altheim added a comment - Okay, can't speak for the weblogic issue, but for Apache/Tomcat 5.5/6 as per recommendation http://www.jspwiki.org/wiki/TomcatAndUTF8 I added the attribute to server.xml as recommended, rebooted tomcat and things are working as expected. Thanks!
          Hide
          Janne Jalkanen added a comment -

          Ugh, obviously without the code-tags above; just mistyped an angle bracket instead of a curly one.

          Show
          Janne Jalkanen added a comment - Ugh, obviously without the code-tags above; just mistyped an angle bracket instead of a curly one.
          Hide
          Janne Jalkanen added a comment -

          Does this discussion help at all?

          http://jira.atlassian.com/browse/JRA-1682

          It proposes modifying weblogic.xml to add UTF-8 encoding, like this

          <code>
          <weblogic-web-app>
          <jsp-descriptor>
          <jsp-param>
          <param-name>encoding</param-name>
          <param-value>UTF-8</param-value>
          </jsp-param>
          <jsp-param>
          <param-name>compilerSupportsEncoding</param-name>
          <param-value>false</param-value>
          </jsp-param>
          </jsp-descriptor>
          </weblogic-web-app>
          </code>

          Show
          Janne Jalkanen added a comment - Does this discussion help at all? http://jira.atlassian.com/browse/JRA-1682 It proposes modifying weblogic.xml to add UTF-8 encoding, like this <code> <weblogic-web-app> <jsp-descriptor> <jsp-param> <param-name>encoding</param-name> <param-value>UTF-8</param-value> </jsp-param> <jsp-param> <param-name>compilerSupportsEncoding</param-name> <param-value>false</param-value> </jsp-param> </jsp-descriptor> </weblogic-web-app> </code>
          Hide
          Jürgen Weber added a comment -

          No, the output is no Chinese, it's rather some kind of not recognizing UTF-8

          Sorry for the double attachment, the upload applet doesn't work too well with FF.

          Anyway, to check:

          byte[] b = "ü".getBytes("utf-8");

          => b =

          { 0xc3, 0xbc }

          So, it looks for some kind of UTF-8 problem.

          Show
          Jürgen Weber added a comment - No, the output is no Chinese, it's rather some kind of not recognizing UTF-8 Sorry for the double attachment, the upload applet doesn't work too well with FF. Anyway, to check: byte[] b = "ü".getBytes("utf-8"); => b = { 0xc3, 0xbc } So, it looks for some kind of UTF-8 problem.
          Hide
          Jürgen Weber added a comment -

          Wrong output of German umlaut u with dots.

          Show
          Jürgen Weber added a comment - Wrong output of German umlaut u with dots.
          Hide
          Jürgen Weber added a comment -

          Wrong output of German umlaut u with dots.

          Show
          Jürgen Weber added a comment - Wrong output of German umlaut u with dots.
          Hide
          Murray Altheim added a comment -

          > I presume you already checked your tomcat connector's
          > settings: http://www.jspwiki.org/wiki/TomcatAndUTF8

          Hi Harry,

          I myself don't have access to theTomcat settings of this particular
          server but have forwarded the link on to the technical support team.
          Will inform if this what it turns out to be.

          Thanks,

          Murray

          Show
          Murray Altheim added a comment - > I presume you already checked your tomcat connector's > settings: http://www.jspwiki.org/wiki/TomcatAndUTF8 Hi Harry, I myself don't have access to theTomcat settings of this particular server but have forwarded the link on to the technical support team. Will inform if this what it turns out to be. Thanks, Murray
          Hide
          Harry Metske added a comment -

          I presume you already checked your tomcat connector's settings: http://www.jspwiki.org/wiki/TomcatAndUTF8

          For WebLogic: can you have a look at JSPWIKI-348, is WebLogic behaving like OC4J ? If it is, then we have to change the patch.

          regards,
          Harry

          Show
          Harry Metske added a comment - I presume you already checked your tomcat connector's settings: http://www.jspwiki.org/wiki/TomcatAndUTF8 For WebLogic: can you have a look at JSPWIKI-348 , is WebLogic behaving like OC4J ? If it is, then we have to change the patch. regards, Harry
          Hide
          Murray Altheim added a comment -

          I've noticed the same thing here. I moved a prototype server running Firefox 3 on Apache/Tomcat 5.5 running on Ubuntu linux, where things were fine, over to a Tomcat (5.5?) running on Solaris 10. We had macron'd vowels in the moved pages that suddenly began displaying incorrectly (e.g., a macron-a as uppercase-umlaut-A with a second boxed error indicator), this occurring both with page content and with page names.

          Moreover, repeatedly editing such pages causes the erroneous characters to gradually increase in length, even upon repeated previews, such that eventually one ends up with a string of dozens of the wrong characters.

          I suspect this may be due to a Tomcat setting on the new server, am still investigating.

          Show
          Murray Altheim added a comment - I've noticed the same thing here. I moved a prototype server running Firefox 3 on Apache/Tomcat 5.5 running on Ubuntu linux, where things were fine, over to a Tomcat (5.5?) running on Solaris 10. We had macron'd vowels in the moved pages that suddenly began displaying incorrectly (e.g., a macron-a as uppercase-umlaut-A with a second boxed error indicator), this occurring both with page content and with page names. Moreover, repeatedly editing such pages causes the erroneous characters to gradually increase in length, even upon repeated previews, such that eventually one ends up with a string of dozens of the wrong characters. I suspect this may be due to a Tomcat setting on the new server, am still investigating.

            People

            • Assignee:
              Harry Metske
              Reporter:
              Jürgen Weber
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development