FormFragment currently "collects" actions but ultimately embeds them in the form itself. It also uses a hidden form field to control whether the actions for the fragment are executed on the server side.
An alternative approach would be for the fragment's actions to be stored in its own hidden field. On a form submit, the fragment could remove the element and, thus, no update to the server for those fields and no need to maintain the "enabled/disabled" form field.
This also has additional applications for use with FormInjector and Ajax forms wherein rows can be easily added and removed from the client-side.