NodeletParser ignore XML encoding.

• Key: IBATIS-349
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Daigo Kobayashi
• Component/s: Core
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0

Environment

Windows XP(Japanese)
jdk 1.5.0_06

Description

NodeletParser ignore XML encoding. Because NodeletParser use Resources, and in some case Resources use ClassLoader#getResourceAsStream with OS default encoding not XML's.

In some environment, for example OS use Shift_JIS and XML use UTF-8, iBatis doesn't work correctly because of broken character.

Comments

[Daigo Kobayashi]

I create patch for this encoding problem.

[Daigo Kobayashi]

I find problem of previous patch. If InputStram is null, that cause NullPointerException. So I add null check code.

[Daigo Kobayashi]

Is it possible to merge attatched code before next release 2.2.1 (or 2.3) ?

[Clinton Begin]

Can you upload the whole file? Not just the patch. Believe it or not, it's much easier to deal with.

[Daigo Kobayashi]

Attach whole file of Resources.java.

[Daigo Kobayashi]

Is there any progress? Attachment is not enough?

[Jeff Butler]

Would it work for you if we added methods to Resources like this:

Resources.getResourceAsReader(String resource, String encoding) {
    return new InputStreamReader(getResourceAsStream(resource), Charset.forName(encoding));
}


Resources.getResourceAsReader(ClassLoader loader, String resource, String encoding) {
    return new InputStreamReader(getResourceAsStream(loader, resource), Charset.forName(encoding));
}


This would be a lot simpler than opening the XML file twice - the first time just to get the encoding.

[Jeff Butler]

Well, my first comment works for the SqlMapConfig file, but doesn't provide any help for the SqlMap files.

Take a look at IBATIS-340 - a similar problem. Of the two solutions, I like the solution proposed in IBATIS-340 better. That's what I'd like to do unless you have some compelling reason that it won't work.

[Daigo Kobayashi]

XML file have it's own encoding. So we should not ignore that encoding. If we follow this solution, we have to set encoding and ignore xml's encoding.

And xml specification says, if encoding is not specified parser must use UTF-8 or UTF-16 and if specified follow it. So this solution violate xml's encoding specification.

[Jeff Butler]

This is a complex issue. The real problem is that iBATIS ignores the encoding issue altogether in parsing - because we only accept a character reader.

I had a quick look at Xerces source to see how they autodetect the encoding and their method is quite different than what you propose. Your method still uses the system default encoding to open the file, a potential source of errors. And it ignores the fact that the real encoding of the file may be different than what's declared in the processing instruction. Xerces figures out the encoding by looking at the first 4 bytes of the file. This is a complex algorithm that we don't want to replicate in iBATIS.

I think the best way to resolve this is to allow iBATIS to accept a byte stream for parsing - then we could leverage the parser's built in support for dealing with different encodings. I'll have a look at doing that.

In the meantime, please try manually setting the encoding to UTF-8 using the fix for IBATIS-340. I'd like to know if this addresses your immediate issue.

[Jeff Butler]

I've committed some changes for IBATIS-373 that allow iBATIS to build the SqlMapClient from an InputStream rather than a Reader. This delegates the encoding issue to the parser - where it should be IMHO. You can use it like this:

String resource = "myconfig/SqlMapConfig.xml";
InputStream inputStream = Resources.getResourceAsStream(resource);
SqlMapClient client = SqlMapClientBuilder.buildSqlMapClient(inputStream);

Please give this a try (or you can wait for 2.3.0 later this week). If one of these alternatives resolves the issue, then I'd like to close this ticket.

Thanks for all the feedback!

[Daigo Kobayashi]

This works fine. And this is ideal solution.
I really appriciate your great job.

BTW, I'm using iBatis with springframework. So I have to fix spring's SqlMapClientFactoryBean.java code. (trivial change.)
So I think some action to spring team is necessary.

Could you report this problem to spring team? Or should I report?

[Jeff Butler]

Great! I'm glad this fix worked.

I think you should open the enhancement request with the Spring team. You could probably provide your modification directly to them, and you're in a better position to test than I am.

[Jeff Butler]

This issue is resolved with the changes for IBATIS-373.

See comments for more details.

[Daigo Kobayashi]

I posted enhancement request .
http://opensource.atlassian.com/projects/spring/browse/SPR-2895