[Linux] std::codecvt_byname("*.UTF-8").in() to_next greater than expected

• Key: STDCXX-435
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Critical
• Assignee: Martin Sebor
• Reporter: Mark Brown
• Component/s: 22. Localization
• Affects Version/s: 4.1.3, 4.1.4, 4.2.0
• Fix Version/s: 4.2.1

Description

When compiled with gcc 4.1.1 on Linux the program below runs successfully to completion as it should. When compiled with stdcxx the facet returns a to_next value that is greater than the number of internal (wchar_t) characters actually produced by the conversion and consequently the program aborts.

$ cat t.cpp && make t && ./t
#include <cassert>
#include <cwchar>
#include <locale>

int main ()
{
    const std::locale utf8 ("en_US.UTF-8");
    typedef std::codecvt<wchar_t, char, std::mbstate_t> UTF8_Cvt;

    const UTF8_Cvt &cvt = std::use_facet<UTF8_Cvt>(utf8);

    const char src[] = "abc";
    wchar_t dst [2] = { L'\0' };

    const char* from_next;

    wchar_t* to_next;

    std::mbstate_t state = std::mbstate_t ();

    const std::codecvt_base::result res =
        cvt.in (state,
                src, src + 1, from_next,
                dst, dst + 2, to_next);

    assert (1 == from_next - src);
    assert (1 == to_next - dst);
    assert ('a' == dst [0]);
}

gcc -c -I/home/mbrown/stdcxx/include/ansi -D_RWSTDDEBUG    -I/home/mbrown/stdcxx/include -I/build/mbrown/stdcxx-gcc-4.1.1-11S/include -I/home/mbrown/stdcxx/examples/include  -pedantic -nostdinc++ -g   -W -Wall -Wcast-qual -Winline -Wshadow -Wwrite-strings -Wno-long-long -Wcast-align   t.cpp
t.cpp: In function 'int main()':
t.cpp:21: warning: unused variable 'res'
gcc t.o -o t  -L/build/mbrown/stdcxx-gcc-4.1.1-11S/lib  -lstd11S -lsupc++ -lm 
t: t.cpp:26: int main(): Assertion `1 == from_next - src' failed.
Aborted

Comments

[Martin Sebor]

This bug is actually what's behind the problem described in STDCXX-333.

[Martin Sebor]

The problem seems to be caused by the fact that in libc mode (i.e., when using the underlying C library) codecvt_byname calls (via __rw_libc_do_in) mbsrtowcs() to convert the source sequence without bothering to make sure it's NUL-terminated. The function attempts to convert the source sequence up until the terminating NUL (or an invalid byte) or until it has produced the requested number of destitation characters. When the destination buffer is large enough for more the number of characters in the source sequence the function just keeps converting past the end.

[Martin Sebor]

This is critical because it affects all UTF-8 files. Scheduled for 4.2.0.

[Martin Sebor]

It's too late to do this in time for 4.2.0. Rescheduled for 4.2.1.

[Martin Sebor]

Added a code tag.

Added an ending code tag.

[Eric Lemings]

Martin's patch for this issue.

[Martin Sebor]

Patch applied in r636534.
Regression test committed in r636553.

Marking as Resolved.
Will merge out to 4.2.1 and close after we confirm that everything looks good in nightly builds.

[Farid Zaripov]

Fix and regression test are merged in 4.2.x branch thus: http://svn.apache.org/viewvc?view=rev&revision=648752

[Martin Sebor]

Added 4.1.3 and 4.2.0 to Affects Version/s.