Currently, code uses a mix of LinkedList and ArrayList - just pick one
Currently, directivesByPath structure is sorted but never used in a sorted way, it only performs remove and add operations, no iteration - use a Set instead of a List for values to support faster remove operation. Use a HashSet instead of a TreeSet for keys since it doesn't appear that order really matters.
The CacheDirective class needs a better hashcode implementation since it will be used in a Set. Do not instantiate a HashBuilder object every time hashcode is called. Ouch.