Some thoughts on the patch (v2):
Does the patch work with sort avoidance? If a scan is against an index and there is an ORDER BY clause that matches the index then we avoid the order by.
This optimization has to be turned off if NULLS FIRST is set (unless the columns are not-nullable).
The passing of an extra argument to the compare() method, and for the regular compare now calling the new compare method will affect performance. I don't know how significant it will be, but within an index btree lookup compare will be called a lot, and now we are passing in an argument which will always be false. A possible alternative that would not affect existing performance, would be to have a single nulls first compare in DataType like (code only, comments you added should be kept):
public final int compare(DataValueDescriptor other, boolean nullsFirst) throws StandardException
if (this.isNull() || other.isNull())
return nullsFirst ? 1 : -1;
return nullsFirst ? -1 : 1;
return 0; // both null
Moving the single argument compare() method in each data type to the parent class DataType would avoid repeated identical code in each class. However with the above comment it would mean this comment doesn't apply, but it's a idea to remember for future changes, if possible push duplicate code up the hierarchy.
For IndexColumnOrder, one approach would be to have two constructors, one with the old signature which would default nulls first to false, and then the new one. This would mean that code that isn't changing doesn't need to be modified by the patch.