Please provide AttributeModifier#remove(String) method which deals with the internals in AttributeModifier.
AttributeModifier has special constants : VALUELESS_ATTRIBUTE_ADD and VALUELESS_ATTRIBUTE_REMOVE which are checked with String#equals(). This disallows to set an attribute with their value. It will be better if they are created with 'new String(...)' and later checked by identity. They are static so there is no chance to be interned during deserialization.