thanks Rakesh, it is good work.
just some comments over your patch.
1) in BookieWatcher, if a bookie server is restarted (maybe by admin guys) from R-O mode, seems there is no logic to remove a bookie from R-O mode to write mode. do I miss something?
2) it would be better to put "R-O" as a constant define, maybe put it in BookieProtocol
3) I don't seem anyone use "roEnsembles" defined in LedgerMetadata.java. so who will use it?
4) the exception is thrown when there is no writable dirs found, so the error message should be improved so admin guys could known what happened. BTW, I am not sure BadLedgerDirsException is good enough to describe such kind of issue, how about NoWritableLedgerDirException?
throw new Bookie.BadLedgerDirsException("Failed to create new log in the ledger dirs : " + list);
+ ArrayList<File> listOfDirs = new ArrayList<File>(list);
How about we keep a list of writable dirs, so we don't need to init a list and remove failed dirs again.
5) in Bookie#addEntryWithRo,
+ zk.setData(myPresenceInZK, "R-O".getBytes(), 0);
seems like it is a conditional set, it might fail if other one did changes on that znode to increment znode version. I am not sure is there any consideration when you using 0 version. why not using -1?
from my side, in future we could provide a tool for admin guys to turn a bookie server into R-O mode to fix some issue then turn it back to W-R mode. so the znode version would be changed outside bookie server, it would be better to using -1 matching any versions of znode.
BTW, it would be better to put a patch to review board (https://reviews.apache.org) if the patch is not simple, which might be convient for reviewing