The SIGSEGV can be solved by doing some safety checks at the beginning of compare: check that offset>=0 and offset+length<=bytes.length. If you use Unsafe, you have to make sure that your parameters are 1000% correct, that's all. This is why java.nio does lots of checks in their Buffer methods.
You also have to copy offset, length and the actual byte reference to a local variable at the beginning and before the bounds checks (because otherwise another thread could change the public non-final fields in BytesRef and cause SIGSEGV). BytesRef is a user-visible class so it must be 100% safe against all usage-violations.
Based on this additional overhead, the whole comparator makes no sense except for terms with a size of 200 bytes. But Lucene terms are in 99% of all cases shorter.
If you want to use this comparator, just subclass Lucene40Codec and return it as term comparator, this can be completely outside Lucene. You can even use Guava.