The maven-release-plugin is pretty much designed to work with a single repository and tag and branch from this. As soon as a project tree is more complex and e.g. uses nested or multiple repositories (such as the Mercurial subrepos), it fails.
The attached patch fixes most of the use cases that allow releasing large (reactor) projects that span multiple projects and use one repository per project.
revertOrder (boolean) - Default: false
Reverts the order in which commit, tag and branch process multi-repos. E.g. in Mercurial, the main repository (which contains the subrepos) must be processed last, because it implicitly records state of the relationship between the main and the sub repository. If it gets committed first, then this state is not recorded correctly. By reverting the order, the main repository is committed last.
commitAllChanges (boolean) - Default: false
The release plugin tries to explicitly list which files it commits. However, in the case of a multi-repository tree, in then tries e.g. to commit repo/pom.xml, repo/a/pom.xml and repo/b/pom.xml in the context of "repo" which then fails (because repo/a/pom.xml and repo/b/pom.xml are actually part of the subrepos a and b). Setting this parameters omits the list of files and tells the SCM to "commit everything". E.g. Mercurial then picks up the changes correctly and also records the implicit state between master and sub repositories correctly.
tagByProject, branchByProject (boolean) - Default: false
Similar to the existing 'commitByProject', these options select whether a tag or a branch should be created by running the tag command in the root of the tree or by looping through all projects and tagging or branching them one-by-one. Default is to tag in the root.
tagRequiresCommit / branchRequiresCommit (boolean) - Default: false
Mercurial manages tags by adding entries to the '.hgtags' file, which is managed implicitly by the SCM. If a subrepository gets tagged as part of a larger, multi-repo project, then the changes must be explicitly committed, else they don't get picked up by the main repository. This sounds more complicated than it actually is, the summary is that "this must be 'true' for Mercurial and probably "false" for everything else.
Those changes should work with the 1.4 SCM provider, but were tested only with the 1.5-SNAPSHOT release. It also benefits from fixing the pushChanges support in the Mercurial provider.
If you want to test drive this patch, you should also be interested in