[XERCESC-1846] We should not let the ICU use a replacement character that we know will result in a document that's not well-formed.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.0, 3.0.0
• Component/s: Utilities
• Environment: ICU 4.0, 3.8; Xerces 3.0, 2.8. Probably will be common problem for earlier versions too.; HP-UX B.11.23 U 9000/800; aCC compier, cc
• Assignee: David N Bertoni
• Reporter: Jan Suchy
• Created: 17/Dec/08 08:17
• Updated: 03/Nov/09 09:48

Description

Jan Suchý wrote:
> Hi Jesse,
> thank you for your answer and ideas.
> I have found one kind of solution to patch the transcoder wrap class:
> src\xercesc\util\Transcoders\ICU\ICUTransService.cpp
>
> adding there to constructor of ICUTranscoder::ICUTranscoder these lines:
>
> UErrorCode uerr = U_ZERO_ERROR;
> ucnv_setSubstChars(toAdopt, "?", 1, &uerr);
> ...
>
> Than, the "?" character is used as replacement char, when using icu.
> This is ICU specific solutions and is not clear, because there is necessary to rebuild xerces lib. I would like to see some possible switch around XMLFormatter class, but there is unknown UConverter form ICU which will be used next, because there is nothing to know which transcoder will be called later.

Please create a Jira issue because this is a bug. We should not let the
ICU use a replacement character that we know will result in a document
that's not well-formed.

Dave

HISTORY:
------------------------------------------------------------------------------------------------------------------------

Hi Jesse,
thank you for your answer and ideas.
I have found one kind of solution to patch the transcoder wrap class:
src\xercesc\util\Transcoders\ICU\ICUTransService.cpp

adding there to constructor of ICUTranscoder::ICUTranscoder these lines:

UErrorCode uerr = U_ZERO_ERROR;
ucnv_setSubstChars(toAdopt, "?", 1, &uerr);
...

Than, the "?" character is used as replacement char, when using icu.
This is ICU specific solutions and is not clear, because there is necessary to rebuild xerces lib. I would like to see some possible switch around XMLFormatter class, but there is unknown UConverter form ICU which will be used next, because there is nothing to know which transcoder will be called later.

Not optimal, but works.
thank you,
jan

> ------------ Původní zpráva ------------
> Od: Jesse Pelton <[e-mail]>
> Předmět: RE: xerces/ICU unicode alias for weak encoding when
> serializing/converting to CP
> Datum: 16.12.2008 15:31:01
> ----------------------------------------
> I'm not an expert on this area, but the transcoders included with Xerces do not
> provide any way to specify the replacement character, and ICU may be the same.
> Even if ICU gives you a way to do so, I'm not sure how you'd get access to a
> transcoder instance to alter.
>
> Note, though, that 0x1A is not a legal character in an XML document. (Oracle's
> parser is correct in rejecting it.) I think it's safe to assume in your scenario
> that any such characters in a serialized document are replacements for
> unrepresentable characters. You should therefore be able to post-process the
> serialization output and replace 0x1A with one or more characters of your
> choosing. If you don't want to post-process the whole document, you could derive
> an XMLFormatTarget that replaces the replacement character in each chunk of data
> handed to it. Neither option is exactly elegant, but I'd probably do the latter;
> it'll work regardless of your format target, where the former approach requires
> serializing to memory.
>
>
> ----Original Message----
> From: Jan Suchý [e-mail]
> Sent: Tuesday, December 16, 2008 5:37 AM
> To: [e-mail]
> Subject: RE: xerces/ICU unicode alias for weak encoding when
> serializing/converting to CP
>
> Hello again,
> i have tried to use class:
>
> http://xerces.apache.org/xerces-c/apiDocs-2/classXMLFormatter.html#_details
>
> with attributes: NoEscapes , UnRep_Replace
>
> and the problematic char was replaced by:
> ^Z
>
> But it is still not solving problem with Oracle DB XML parser to parse this xml.
> I have got this error:
>
> ORA-31011: XML parsing failed
> ORA-19202: Error occurred in XML processing
> LPX-00216: invalid character 26 (0x1A)
> Error at line 22
>
> I would like to replace unknown character with my own character, which will be
> parseable (for example char "?" or "_").
> How can I change replacement character, which is used as default?
>
> Thank anybody for any idea.
>
> Have a nice day,
> Jan
>
>
> > ------------ Původní zpráva ------------
> > Od: Jan Suchý <[e-mail]>
> > Předmět: RE: xerces/ICU unicode alias for weak encoding when
> > serializing/converting to CP
> > Datum: 16.12.2008 09:35:40
> > ----------------------------------------
> > Hello Jesse,
> > thank you for your answer it seems to be promising. I'll look at it.
> > Jan
> >
> >
> > > ------------ Původní zpráva ------------
> > > Od: Jesse Pelton <[e-mail]>
> > > Předmět: RE: xerces/ICU unicode alias for weak encoding when
> > > serializing/converting to CP
> > > Datum: 15.12.2008 18:15:49
> > > ----------------------------------------
> > > The constructors for the Xerces XMLFormatter object all take an UnRepFlags
> > > argument that allows you to specify how to handle unrepresentable
> characters.
> >
> > > So does XMLFormatter::formatBuf(). It appears that the transcoder gets to
> > > decide what character to replace unrepresentable characters with.
> > >
> > > Hope that helps.
> > >
> > > ----Original Message----
> > > From: Jan Suchý [e-mail]
> > > Sent: Monday, December 15, 2008 4:25 AM
> > > To: [e-mail]
> > > Subject: xerces/ICU unicode alias for weak encoding when
> > serializing/converting
> > > to CP
> > >
> > > Hello all,
> > > I need to obtain output XML in iso-8859-2 encoding.
> > > I am using UTF-8 as input encoding.
> > > There is some character, in UTF-8 xml, which is not representable in
> > > iso-8859-2.
> > > I am using ICU 3.8, xerces 2.8 and Xqilla svn 702.
> > >
> > > After serializing XML to iso-8859-2 the problematic character is serialized
> > by
> > > ICU/xerces/xq to:
> > >
> > > –
> > >
> > > The problem is, that if I will send message in iso-8859-2 with character
> > > – inside to Oracle DB, the Oracle parser
> > >
> > > does not like this character and this error is obtained:
> > >
> > > ORA-31011: XML parsing failed, LPX-00217: invalid character 8211 (U+2013)
> > >
> > > So, what I am looking for is some method, how to say to the ICU or to
> Xerces
> > or
> > > to XQ, that the Unicode character, must
> > >
> > > not be included in result and must be for example replaced by character
> "?",
> > to
> > > avoid Oracle parser to process it.
> > >
> > > I would like to find clear solution, like saying to ICU not calling
> callback
> > > function or define own alias or behavior on
> > >
> > > this situation. Is it possible?
> > > Any ideas?
> > > Thank you
> > > Jan Suchy