Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Core
    • Labels:
      None
    • Environment:
      Windows XP(Japanese)
      jdk 1.5.0_06

      Description

      NodeletParser ignore XML encoding. Because NodeletParser use Resources, and in some case Resources use ClassLoader#getResourceAsStream with OS default encoding not XML's.

      In some environment, for example OS use Shift_JIS and XML use UTF-8, iBatis doesn't work correctly because of broken character.

      1. Resources.patch
        3 kB
        Daigo Kobayashi
      2. Resources.patch
        3 kB
        Daigo Kobayashi
      3. Resources.java
        11 kB
        Daigo Kobayashi

        Activity

        Hide
        Daigo Kobayashi added a comment -
        Show
        Daigo Kobayashi added a comment - I posted enhancement request . http://opensource.atlassian.com/projects/spring/browse/SPR-2895
        Hide
        Jeff Butler added a comment -

        This issue is resolved with the changes for IBATIS-373.

        See comments for more details.

        Show
        Jeff Butler added a comment - This issue is resolved with the changes for IBATIS-373 . See comments for more details.
        Hide
        Jeff Butler added a comment -

        Great! I'm glad this fix worked.

        I think you should open the enhancement request with the Spring team. You could probably provide your modification directly to them, and you're in a better position to test than I am.

        Show
        Jeff Butler added a comment - Great! I'm glad this fix worked. I think you should open the enhancement request with the Spring team. You could probably provide your modification directly to them, and you're in a better position to test than I am.
        Hide
        Daigo Kobayashi added a comment -

        This works fine. And this is ideal solution.
        I really appriciate your great job.

        BTW, I'm using iBatis with springframework. So I have to fix spring's SqlMapClientFactoryBean.java code. (trivial change.)
        So I think some action to spring team is necessary.

        Could you report this problem to spring team? Or should I report?

        Show
        Daigo Kobayashi added a comment - This works fine. And this is ideal solution. I really appriciate your great job. BTW, I'm using iBatis with springframework. So I have to fix spring's SqlMapClientFactoryBean.java code. (trivial change.) So I think some action to spring team is necessary. Could you report this problem to spring team? Or should I report?
        Hide
        Jeff Butler added a comment -

        I've committed some changes for IBATIS-373 that allow iBATIS to build the SqlMapClient from an InputStream rather than a Reader. This delegates the encoding issue to the parser - where it should be IMHO. You can use it like this:

        String resource = "myconfig/SqlMapConfig.xml";
        InputStream inputStream = Resources.getResourceAsStream(resource);
        SqlMapClient client = SqlMapClientBuilder.buildSqlMapClient(inputStream);

        Please give this a try (or you can wait for 2.3.0 later this week). If one of these alternatives resolves the issue, then I'd like to close this ticket.

        Thanks for all the feedback!

        Show
        Jeff Butler added a comment - I've committed some changes for IBATIS-373 that allow iBATIS to build the SqlMapClient from an InputStream rather than a Reader. This delegates the encoding issue to the parser - where it should be IMHO. You can use it like this: String resource = "myconfig/SqlMapConfig.xml"; InputStream inputStream = Resources.getResourceAsStream(resource); SqlMapClient client = SqlMapClientBuilder.buildSqlMapClient(inputStream); Please give this a try (or you can wait for 2.3.0 later this week). If one of these alternatives resolves the issue, then I'd like to close this ticket. Thanks for all the feedback!
        Hide
        Jeff Butler added a comment -

        This is a complex issue. The real problem is that iBATIS ignores the encoding issue altogether in parsing - because we only accept a character reader.

        I had a quick look at Xerces source to see how they autodetect the encoding and their method is quite different than what you propose. Your method still uses the system default encoding to open the file, a potential source of errors. And it ignores the fact that the real encoding of the file may be different than what's declared in the processing instruction. Xerces figures out the encoding by looking at the first 4 bytes of the file. This is a complex algorithm that we don't want to replicate in iBATIS.

        I think the best way to resolve this is to allow iBATIS to accept a byte stream for parsing - then we could leverage the parser's built in support for dealing with different encodings. I'll have a look at doing that.

        In the meantime, please try manually setting the encoding to UTF-8 using the fix for IBATIS-340. I'd like to know if this addresses your immediate issue.

        Show
        Jeff Butler added a comment - This is a complex issue. The real problem is that iBATIS ignores the encoding issue altogether in parsing - because we only accept a character reader. I had a quick look at Xerces source to see how they autodetect the encoding and their method is quite different than what you propose. Your method still uses the system default encoding to open the file, a potential source of errors. And it ignores the fact that the real encoding of the file may be different than what's declared in the processing instruction. Xerces figures out the encoding by looking at the first 4 bytes of the file. This is a complex algorithm that we don't want to replicate in iBATIS. I think the best way to resolve this is to allow iBATIS to accept a byte stream for parsing - then we could leverage the parser's built in support for dealing with different encodings. I'll have a look at doing that. In the meantime, please try manually setting the encoding to UTF-8 using the fix for IBATIS-340 . I'd like to know if this addresses your immediate issue.
        Hide
        Daigo Kobayashi added a comment -

        XML file have it's own encoding. So we should not ignore that encoding. If we follow this solution, we have to set encoding and ignore xml's encoding.

        And xml specification says, if encoding is not specified parser must use UTF-8 or UTF-16 and if specified follow it. So this solution violate xml's encoding specification.

        Show
        Daigo Kobayashi added a comment - XML file have it's own encoding. So we should not ignore that encoding. If we follow this solution, we have to set encoding and ignore xml's encoding. And xml specification says, if encoding is not specified parser must use UTF-8 or UTF-16 and if specified follow it. So this solution violate xml's encoding specification.
        Hide
        Jeff Butler added a comment -

        Well, my first comment works for the SqlMapConfig file, but doesn't provide any help for the SqlMap files.

        Take a look at IBATIS-340 - a similar problem. Of the two solutions, I like the solution proposed in IBATIS-340 better. That's what I'd like to do unless you have some compelling reason that it won't work.

        Show
        Jeff Butler added a comment - Well, my first comment works for the SqlMapConfig file, but doesn't provide any help for the SqlMap files. Take a look at IBATIS-340 - a similar problem. Of the two solutions, I like the solution proposed in IBATIS-340 better. That's what I'd like to do unless you have some compelling reason that it won't work.
        Hide
        Jeff Butler added a comment -

        Would it work for you if we added methods to Resources like this:

        Resources.getResourceAsReader(String resource, String encoding)

        { return new InputStreamReader(getResourceAsStream(resource), Charset.forName(encoding)); }

        Resources.getResourceAsReader(ClassLoader loader, String resource, String encoding)

        { return new InputStreamReader(getResourceAsStream(loader, resource), Charset.forName(encoding)); }

        This would be a lot simpler than opening the XML file twice - the first time just to get the encoding.

        Show
        Jeff Butler added a comment - Would it work for you if we added methods to Resources like this: Resources.getResourceAsReader(String resource, String encoding) { return new InputStreamReader(getResourceAsStream(resource), Charset.forName(encoding)); } Resources.getResourceAsReader(ClassLoader loader, String resource, String encoding) { return new InputStreamReader(getResourceAsStream(loader, resource), Charset.forName(encoding)); } This would be a lot simpler than opening the XML file twice - the first time just to get the encoding.
        Hide
        Daigo Kobayashi added a comment -

        Is there any progress? Attachment is not enough?

        Show
        Daigo Kobayashi added a comment - Is there any progress? Attachment is not enough?
        Hide
        Daigo Kobayashi added a comment -

        Attach whole file of Resources.java.

        Show
        Daigo Kobayashi added a comment - Attach whole file of Resources.java.
        Hide
        Clinton Begin added a comment -

        Can you upload the whole file? Not just the patch. Believe it or not, it's much easier to deal with.

        Show
        Clinton Begin added a comment - Can you upload the whole file? Not just the patch. Believe it or not, it's much easier to deal with.
        Hide
        Daigo Kobayashi added a comment -

        Is it possible to merge attatched code before next release 2.2.1 (or 2.3) ?

        Show
        Daigo Kobayashi added a comment - Is it possible to merge attatched code before next release 2.2.1 (or 2.3) ?
        Hide
        Daigo Kobayashi added a comment -

        I find problem of previous patch. If InputStram is null, that cause NullPointerException. So I add null check code.

        Show
        Daigo Kobayashi added a comment - I find problem of previous patch. If InputStram is null, that cause NullPointerException. So I add null check code.
        Hide
        Daigo Kobayashi added a comment -

        I create patch for this encoding problem.

        Show
        Daigo Kobayashi added a comment - I create patch for this encoding problem.

          People

          • Assignee:
            Unassigned
            Reporter:
            Daigo Kobayashi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development