I'm not familiar with the various considerations that were made with StandardTokenizer, but please allow me to share some comments anyway.
Perhaps it's useful to distinguish between analysis for information retrieval and analysis for information extraction here?
I like Michael's and Steven's idea of doing tokenization that doesn't discard any information. This is certainly useful in the case of information extraction. For example, if we'd like to extract noun-phrases based on part-of-speech tags, we don't want to conjoin tokens in case there's a punctuation character between two nouns (unless that punctuation character is a middle dot).
Robert is of course correct that we generally don't want to index punctuation characters that occur in every document, so from an information retrieval point of view, we'd like punctuation characters removed.
If there's an established convention that Tokenizer variants discards punctuation and produces the terms that are meant to be directly searchable, it sounds like a good idea that we stick to the convention here as well.
If there's no established convention, it seems useful that a Tokenizer would provide as much details as possible with text being input and leave downstream Filters/Analyzers to remove whatever is suitable based on a particular processing purpose. We can provide common ready-to-use Analyzers with reasonable defaults that users can look to, i.e. to process a specific language or do another common high-level task with text.
Hence, perhaps each Tokenizer can decide what makes the most sense to do based on that particular tokenizer's scope of processing?
To Roberts point, this would leave processing totally arbitrary and consistent, but this would be by design as it wouldn't be Tokenizer's role to enforce any overall consistency – i.e. with regards to punctuation – higher level Analyzers would provide that.