This is really confusing, see
LUCENE-4176 where a user hit this.
The example code from the user there reads:
Its really trappy that this does not invoke the analyzer: because historically we did this unless you specified NOT_ANALYZED.
So i think fieldType.setTokenized(true) should be the default: things like StringField can turn it off.