I'm a little confused about the use of the tokenizer (i have no problems technically, its maybe a naming issue?)
Is this intended for tokenizing file pathnames as its name would suggest? In this case I think the path should have positions, e.g. /foo/bar/whatever.txt is foo(1), bar(1), whatever.txt(1)?
It seems instead, this one is intended for representing hierarchies, as it creates synonyms of /foo, /foo/bar, /foo/bar/whatever.txt... with position increments of zero.
I guess I'm just being picky about naming, but i think this hierarchical case is more specific than 'tokenizing file pathnames' and maybe a name like HierarchyTokenizer (this one too probably isn't the best!) would better represent what it does?