Here's a patch... I added a simple test.

I'm sure it can be beautified etc.

JFlex generates fine, everything compiles, all tests pass.

If we add a target in modules/analysis/icu/build.xml to run GenerateJFlexSupplementaryMacros#main(), maybe named gen-stdtok-supp-macros, the jflex target in modules/analysis/common/build.xml could use a <subant> to call it and auto-generate SUPPLEMENTARY.jflex-macro, no?

If we add a target in modules/analysis/icu/build.xml to run GenerateJFlexSupplementaryMacros#main(), maybe named gen-stdtok-supp-macros, the jflex target in modules/analysis/common/build.xml could use a <subant> to call it and auto-generate SUPPLEMENTARY.jflex-macro, no?

Yeah, i think we could do something like this. We could also consolidate tools, because in general i would rather all the analyzers
be consolidated, they are only split up due to dependencies/large files etc. But tools are different, its just to assist the build.

New patch, with the following changes:

1. Added a new target gen-uax29-supp-macros to modules/analysis/icu/build.xml, and a <subant> call to it from the jflex task in modules/analysis/common/build.xml.
2. Included SUPPLEMENTARY.jflex-macro in UAX29URLEmailTokenizer.jflex in the same way as it is included in StandardTokenizer.jflex
3. Copied the simple supplementary characters test from TestStandardAnalyzer.java to TestUAX29URLEmailTokenizer.java.
4. Modified the CHANGES.txt entry for the UAX#29 issues to include a reference to this issue.

All tests pass.

Removed the WARNING from the UAX29URLEmailTokenizer class javadocs about Unicode supplementary character non-coverage.

We could also consolidate tools, because in general i would rather all the analyzers be consolidated, they are only split up due to dependencies/large files etc. But tools are different, its just to assist the build.

How far would you go with this tools consolidation? All tools across the whole of Scenolunr? Or just the ones under modules/analysis/?

How far would you go with this tools consolidation? All tools across the whole of Scenolunr? Or just the ones under modules/analysis/?

I just meant under the analyzers module... but lets leave this be, i also forgot we have no analyzers module in 3.x.

i think we should commit your latest patch.

i think we should commit your latest patch.

OK, I'll commit shortly.

Committed to trunk: r1055877, branch_3x: r1055904.

Thanks for taking care of this!

I think the added files need svn:eol-style=native ?
Also, I think we should add an ASL2 license to the generated macros?
I noticed the TLD generator does this, but I forgot to do it here.

I think the added files need svn:eol-style=native ?
Also, I think we should add an ASL2 license to the generated macros?
I noticed the TLD generator does this, but I forgot to do it here.

Done: trunk: r1056014, branch_3x: r1056030

Bulk close for 3.1