No problem Robert, thanks for taking a look.
About back-compat: none of the JFlex-based tokenizers on trunk have version-based behavior at this point, in contrast to branch_4x. It could be argued that that was because all previous back-compat version were for 3.X, but this issue introduced a 4.0 version, which puts it within the version X-1 window for trunk/5.0. Should I forward-port the 4.0 back-compat stuff from branch_4x for StandardTokenizer and UAX29URLEmailTokenizer? There are other analysis components on trunk that do different things based on version, so clearly the practice has not been abandoned on trunk.