Patch that adds functionality to the setters to keep the attributes map up to date.
I decided to go with that route because removing groups and payload field involves converting types in a method that is probably used a lot, as the groups/payloads are stored as arrays in the attribute map and as sets in the individual fields.
Test uploaded in this issue passes with this patch applied.
Of course, if you prefer to fix it any other way, just ignore this patch.
Btw, I have another patch to fix implicit groups lined up for when this issue is closed