Proposed implementation and test

If you could tolerate that these fingerprints are not be reliable identifiers of your input, I'm wondering that we could make it more efficient by just using a hash function that doesn't depend on the order of its inputs?

Otherwise this looks rather good to me. Instead of taking the min offset and the max offset as offsets for the final token, I'm wondering that it might make more sense to use 0 and the final offset (the one returned after end() has been called) instead so that we don't treat token chars differently depending on whether they appear before/after the tokens or in the middle? By the way even with the current approach, we don't need to call Math.min/max: As tokens are supposed to be emitted in order, the start offset would be the start offset of the first token and the end offset would be the end offset of the last token.

Thanks for taking a look, Adrien.
Added a v2 patch with following changes:

1) added call to input.end() to get final offset state
2) final state is retained using captureState()
3) Added a FingerprintFilterFactory class

As for the alternative hashing idea :
For speed reasons this would be a nice idea but reduces the read-ability of results if you want to debug any collisions or otherwise display connections.

For compactness reasons (storing in doc values etc) it would always be possible to chain a conventional hashing algo in a TokenFilter on the end of this text-normalizing filter. (Do we already have a conventional hashing TokenFilter?)

+      if (item instanceof char[]) {
+        sb.append((char[]) item);
+      } else {
+        sb.append(item);
+      }

Can we get something else than a char[]? If no I think we should just forcefully cast?

Otherwise +1.

Updated patch - removed instanceof check and added entry to Changes.txt.

Will commit to trunk and 5.x in a day or two if there's no objections

Some final tweaks:
1) Found a bug where separator not appended if first token is length ==1
2) Randomized testing identified issue with input.end() not being called when IOExceptions occur
3) Added missing SPI entry for FingerprintFilterFactory and associated test class for FingerprintFilterFactory

Commit 1698145 from mharwood@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1698145 ]

LUCENE-6747: FingerprintFilter is a new TokenFilter that outputs a single token which is a concatenation of the sorted and de-duplicated set of input tokens.

Commit 1698188 from mharwood@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1698188 ]

LUCENE-6747: FingerprintFilter is a new TokenFilter that outputs a single token which is a concatenation of the sorted and de-duplicated set of input tokens.

Commited to trunk and 5.x

Mark, I see this issue under "Lucene 6.0.0" in the CHANGES.txt file on trunk, should it be moved to 5.4.0?