Is there a way we can write a temp file and do a mv to rename/overwrite replication.properties
Nothing great. Java 7 gives us an atomic move that can overwrite an existing file, but it's impl dependent on if that is supported and it wouldn't work for the arbitrary FileSystem's we support. We would still need some start up logic that could address a crashed state.
Alternate solution would be to keep appending to existing file and read the latest stats from the file.
The problem is we use this same strategy for index.properties which is not so straightforward to do this way.
I think we can simply delete the exist replication.properties before write a new one.
That is the easy fix I mention above, but it's fragile, and like index.properties, not robust in a crash.