Details
Bug
Trivial
Description
Seems like invalid utf-8 sometimes gets passed to Bytesref.utf8ToString() in the verbose "println"s.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Hmm, I don't think this is quite right: in the BYTE1 case, these are the bytes from the term, and we shouldn't pretend they are unicode code points (which is what UnicodeUtil.newString is given).
Ie, we really do need the inputMode to be passed to inputToString.
Really, this test pretends a term is always a utf8 byte sequence, which in general is not the case (terms are arbitrary byte[]), it's just that this test only ever operates on terms that are in fact utf8 byte sequences (I think?).
Indeed I'm also hitting AIOOBE (ant test-core -Dtestcase=TestFSTs -Dtestmethod=testRandomWords -Dtests.seed=-3451527662631579719:-3355372777860187201):
There was 1 failure: 1) testRandomWords(org.apache.lucene.util.fst.TestFSTs) java.lang.ArrayIndexOutOfBoundsException: 44 at org.apache.lucene.util.UnicodeUtil.UTF8toUTF16(UnicodeUtil.java:586) at org.apache.lucene.util.BytesRef.utf8ToString(BytesRef.java:203) at org.apache.lucene.util.fst.TestFSTs.inputToString(TestFSTs.java:989) at org.apache.lucene.util.fst.TestFSTs.access$000(TestFSTs.java:53) at org.apache.lucene.util.fst.TestFSTs$FSTTester.verifyPruned(TestFSTs.java:833) at org.apache.lucene.util.fst.TestFSTs$FSTTester.doTest(TestFSTs.java:507) at org.apache.lucene.util.fst.TestFSTs$FSTTester.doTest(TestFSTs.java:366) at org.apache.lucene.util.fst.TestFSTs.doTest(TestFSTs.java:214) at org.apache.lucene.util.fst.TestFSTs.testRandomWords(TestFSTs.java:963) at org.apache.lucene.util.fst.TestFSTs.testRandomWords(TestFSTs.java:938)
Spooky because this test supposedly creates random valid unicode strings (_TestUtil.randomRealisticUnicodeString)... hmmm.
Spooky because this test supposedly creates random valid unicode strings (_TestUtil.randomRealisticUnicodeString)... hmmm.
but then it breaks them down into prefixes and those aren't always valid utf-8...
Ahhh that's what makes the invalid UTF8. OK. Can we just change that one place (that cuts a potentially invalid UTF8 prefix) to just use BytesRef.toString?
Wherever the test is printing out a term prefix, it just calls IntsRef.toString() rather than try to convert this to something more human-readable.
Looks great James, thanks! I confirmed this fixes the above exc when I run w/ verbose. I'll commit shortly – I just changed the new arg's name to "isValidUnicode", and fixed up the whitespace.
Thanks James!
this just calls UnicodeUtil.newString(..) instead of BytesRef.utf8ToString() in all cases.