[CODEC-127] Non-ascii characters in source files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5
• Fix Version/s: 1.6
• Assignee: Unassigned
• Reporter: Sebb
• Created: 13/Aug/11 11:45
• Updated: 10/Nov/11 18:10
• Resolved: 17/Aug/11 16:32

Description

Some of the test cases include characters in a native encoding (possibly UTF-8), rather than using Unicode escapes.

This can cause a problem for IDEs if they don't know the encoding (e.g. cause compilation errors, which is how I found the issue), and possibly some transformations may corrupt the contents, e.g. fixing EOL.

I think we should have a rule of using Unicode escapes for all such non-ascii characters.
It's particularly important for non-ISO-8859-1 characters.

Some example classes with non-ascii characters:

binary\Base64Test.java:96         byte[] decode = b64.decode("SGVsbG{´┐¢´┐¢´┐¢´┐¢´┐¢´┐¢}8gV29ybGQ=");
language\ColognePhoneticTest.java:110             {"m├Ânchengladbach", "664645214"},
language\ColognePhoneticTest.java:130         String[][] data = {{"bergisch-gladbach", "174845214"}, {"M├╝ller-L├╝denscheidt", "65752682"}};
language\ColognePhoneticTest.java:137             {"Meyer", "M├╝ller"},
language\ColognePhoneticTest.java:143             {"ganz", "Gänse"},
language\DoubleMetaphoneTest.java:1222         this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "S");
language\DoubleMetaphoneTest.java:1227         this.getDoubleMetaphone().isDoubleMetaphoneEqual("´┐¢", "N");
language\SoundexTest.java:367         if (Character.isLetter('´┐¢')) {
language\SoundexTest.java:369                 Assert.assertEquals("´┐¢000", this.getSoundexEncoder().encode("´┐¢"));
language\SoundexTest.java:375             Assert.assertEquals("", this.getSoundexEncoder().encode("´┐¢"));
language\SoundexTest.java:387         if (Character.isLetter('´┐¢')) {
language\SoundexTest.java:389                 Assert.assertEquals("´┐¢000", this.getSoundexEncoder().encode("´┐¢"));
language\SoundexTest.java:395             Assert.assertEquals("", this.getSoundexEncoder().encode("´┐¢"));

The characters are probably not correct above, because I used a crude perl script to find them:

perl -ne "$.=1 if $s ne $ARGV;print qq($ARGV:$. $_) if m/\P{ASCII}/;$s=$ARGV;" xxxx.java

language\SoundexTest.java:367 in particular is incorrect, because it's supposed to be a single character.

Now one might think that native2ascii -encoding UTF-8 would fix that, but it gives:

if (Character.isLetter('\ufffd'))

which is an "unknown" character.

Similarly for binary\Base64Test.java:96.

It's not all that clear what the Unicode escapes should be in these cases, but probably not the unknown character.

[Possibly the characters got mangled at some point, or maybe they have always been wrong]

The ColognePhoneticTest.java cases are less serious, as the characters are valid ISO-8859-1 (accented German), but given that the rest of the file uses unicode escaps, I think they should be changed too (but add comments to say what they are, e.g. o-umlaut, u-umlaut)