Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3, 2.0.0
    • Fix Version/s: 1.3
    • Component/s: Replication
    • Labels:
    • Skill Level:
      Regular Contributors Level (Easy to Medium)

      Description

      patch to move couch replication modules in in a standalone couch_replicator application. It's also available on my github:

      https://github.com/benoitc/couchdb/compare/master...couch_replicator

        Issue Links

          Activity

          Hide
          Benoit Chesneau added a comment -

          fix httpc_pool tests. All tests are now working.

          Show
          Benoit Chesneau added a comment - fix httpc_pool tests. All tests are now working.
          Hide
          Filipe Manana added a comment -

          Thanks Benoit.

          I tested the second patch, but test 002 is failing:

          fdmanana 14:00:20 ~/git/hub/couchdb4 (master)> ./test/etap/run -v src/couch_replicator/test/002-replication-compact.t
          src/couch_replicator/test/002-replication-compact.t ..

          1. Current time local 2011-10-30 14:00:25
          2. Using etap version "0.3.4"
            1..376
            Apache CouchDB 0.0.0 (LogLevel=info) is starting.
            Apache CouchDB has started. Time to relax.
            [info] [<0.2.0>] Apache CouchDB has started on http://127.0.0.1:59045/
            ok 1 - Source database is idle before starting replication
            ok 2 - Target database is idle before starting replication
          3. Test died abnormally: {'EXIT',
            {noproc,
            {gen_server,call,
            [couch_rep_sup,
            {start_child,
            {"e0ce0ffa71ae1915f953eb8d2f93e348+continuous",
            {gen_server,start_link,
            [couch_replicator,
            {rep,
            {"e0ce0ffa71ae1915f953eb8d2f93e348",

          Good work

          Show
          Filipe Manana added a comment - Thanks Benoit. I tested the second patch, but test 002 is failing: fdmanana 14:00:20 ~/git/hub/couchdb4 (master)> ./test/etap/run -v src/couch_replicator/test/002-replication-compact.t src/couch_replicator/test/002-replication-compact.t .. Current time local 2011-10-30 14:00:25 Using etap version "0.3.4" 1..376 Apache CouchDB 0.0.0 (LogLevel=info) is starting. Apache CouchDB has started. Time to relax. [info] [<0.2.0>] Apache CouchDB has started on http://127.0.0.1:59045/ ok 1 - Source database is idle before starting replication ok 2 - Target database is idle before starting replication Test died abnormally: {'EXIT', {noproc, {gen_server,call, [couch_rep_sup, {start_child, {"e0ce0ffa71ae1915f953eb8d2f93e348+continuous", {gen_server,start_link, [couch_replicator, {rep, {"e0ce0ffa71ae1915f953eb8d2f93e348", Good work
          Hide
          Benoit Chesneau added a comment - - edited

          Thanks Filipe. Odd. I didn't reproduce that on my machine. Anyway third attempt. This one remove couch_rep_sup from couch.app and register couch_replicator_tasks_sup . All tests pass on this machine.

          Show
          Benoit Chesneau added a comment - - edited Thanks Filipe. Odd. I didn't reproduce that on my machine. Anyway third attempt. This one remove couch_rep_sup from couch.app and register couch_replicator_tasks_sup . All tests pass on this machine.
          Hide
          Filipe Manana added a comment -

          All tests now pass except test/etap/001-load.t which needs some lines updated with the new module names.
          From my side, after that small change, it's ready to go into master.

          Thanks for doing this change Benoît.

          Show
          Filipe Manana added a comment - All tests now pass except test/etap/001-load.t which needs some lines updated with the new module names. From my side, after that small change, it's ready to go into master. Thanks for doing this change Benoît.
          Hide
          Benoit Chesneau added a comment -

          4th iteration of the patch.

          • fix 001-load.t,
          • add 01-load.t to couch_replicator
          • rename 00*.t to 0*.t in couch_replicator tests, we probably won't have more than 99 tests here.
          Show
          Benoit Chesneau added a comment - 4th iteration of the patch. fix 001-load.t, add 01-load.t to couch_replicator rename 00*.t to 0*.t in couch_replicator tests, we probably won't have more than 99 tests here.
          Hide
          Filipe Manana added a comment -

          All tests are now passing for me.
          Just noticed that src/couch_replicator/ebin/couch_replicator.app should be added to .gitignore as wel, since it's modified after every build.

          +1 to go to master

          Show
          Filipe Manana added a comment - All tests are now passing for me. Just noticed that src/couch_replicator/ebin/couch_replicator.app should be added to .gitignore as wel, since it's modified after every build. +1 to go to master
          Hide
          Adam Kocoloski added a comment -

          Thanks for doing this, Benoit, it seems like a really good idea. I have a few questions from reading the patch:

          • Why is couch_replicator.app committed to version control? The app.src should be sufficient.
          • The list of registered processes in the app.src file is incomplete. At the very least couch_replicator_manager belongs in there.
          • I have to say it: our application startup procedure is a huge mess. This patch doesn't necessarily make matters any worse in that regard, but unfortunately it doesn't improve matters either. It starts the top-level supervisor of the couch_replicator application from inside the couch application, but never actually starts the couch_replicator application. You and I both know what the Right Way looks like at this point, hopefully we'll bring Apache CouchDB up to snuff soon. The current patch might be OK, though if I'm reading it correctly application:start(couch_replicator) will crash if the couch application is already running. Yuck.
          • Are the record definitions in couch_replicator.hrl meant for use by other OTP applications? If not I think we should move the .hrl file to src/. Reserving include/ for headers needed to use the application is a good practice.
          • Nitpick - I prefer to name children in the supervision tree after the callback module. I made the mistake of calling the old child couch_replication_supervisor instead of couch_rep_sup a long time ago and it's bugged me ever since. Can we rename the new child to couch_replicator_tasks_sup instead?
          • Now that the replicator is shaped a little more like an OTP application it bugs me that couch_replicator is both the API module and a gen_server. I don't want to see it changed in this commit, but down the line I'd like to split that out so we have a different module that actually manages the various processes associated with a replication task.

          Again, this is a big step in the right direction. Thanks for getting it going.

          Show
          Adam Kocoloski added a comment - Thanks for doing this, Benoit, it seems like a really good idea. I have a few questions from reading the patch: Why is couch_replicator.app committed to version control? The app.src should be sufficient. The list of registered processes in the app.src file is incomplete. At the very least couch_replicator_manager belongs in there. I have to say it: our application startup procedure is a huge mess. This patch doesn't necessarily make matters any worse in that regard, but unfortunately it doesn't improve matters either. It starts the top-level supervisor of the couch_replicator application from inside the couch application, but never actually starts the couch_replicator application. You and I both know what the Right Way looks like at this point, hopefully we'll bring Apache CouchDB up to snuff soon. The current patch might be OK, though if I'm reading it correctly application:start(couch_replicator) will crash if the couch application is already running. Yuck. Are the record definitions in couch_replicator.hrl meant for use by other OTP applications? If not I think we should move the .hrl file to src/. Reserving include/ for headers needed to use the application is a good practice. Nitpick - I prefer to name children in the supervision tree after the callback module. I made the mistake of calling the old child couch_replication_supervisor instead of couch_rep_sup a long time ago and it's bugged me ever since. Can we rename the new child to couch_replicator_tasks_sup instead? Now that the replicator is shaped a little more like an OTP application it bugs me that couch_replicator is both the API module and a gen_server. I don't want to see it changed in this commit, but down the line I'd like to split that out so we have a different module that actually manages the various processes associated with a replication task. Again, this is a big step in the right direction. Thanks for getting it going.
          Hide
          Benoit Chesneau added a comment -

          I've upgraded my branch with latest changes in replication. Following are my answers:

          • couch_replicator.app is commited? Shouldn't be the case now.
          • couch_replicator_manager is now registered . I'm not sure why we need to register others here. What would be the benefit.
          • About records, I'm not sure if it could be usable actually for other things. However, that is something we could change easily in another patch I think once we are sure it couldn't be used in another parts?
          • nitpick, fixed.
          • I agree, more generally we could have such things in couch_core, for cache support as well imo.

          Side points. I think we could benefit here by using a "simple_one_for_one" supervisor for replication tasks rather a "one_for_one" . So it would simplify the calll and termination handling imo. We are in a case where this kind of supervision works pretty well. In near future I really would like couchdb built as a full erlang release where couch core and others are applications booted on start. Lot of things would benefit from it.

          Show
          Benoit Chesneau added a comment - I've upgraded my branch with latest changes in replication. Following are my answers: couch_replicator.app is commited? Shouldn't be the case now. couch_replicator_manager is now registered . I'm not sure why we need to register others here. What would be the benefit. About records, I'm not sure if it could be usable actually for other things. However, that is something we could change easily in another patch I think once we are sure it couldn't be used in another parts? nitpick, fixed. I agree, more generally we could have such things in couch_core, for cache support as well imo. Side points. I think we could benefit here by using a "simple_one_for_one" supervisor for replication tasks rather a "one_for_one" . So it would simplify the calll and termination handling imo. We are in a case where this kind of supervision works pretty well. In near future I really would like couchdb built as a full erlang release where couch core and others are applications booted on start. Lot of things would benefit from it.
          Hide
          Benoit Chesneau added a comment -

          Ifit's OK I will commit that change later today.

          Show
          Benoit Chesneau added a comment - Ifit's OK I will commit that change later today.
          Hide
          Adam Kocoloski added a comment -

          Is the link still https://github.com/benoitc/couchdb/compare/master...couch_replicator ? I see a new commit from you but going through the checklist I see that couch_replicator.app is still being committed.

          Show
          Adam Kocoloski added a comment - Is the link still https://github.com/benoitc/couchdb/compare/master...couch_replicator ? I see a new commit from you but going through the checklist I see that couch_replicator.app is still being committed.
          Hide
          Benoit Chesneau added a comment -

          Thanks Adam, not sure why it has been committed at first... Anyway I've committed a new change :

          https://github.com/benoitc/couchdb/commit/2df4351cbfb28f543876377ba82be30175a95fff

          which fix that issue. If this is ok i will merge the changes and commit it to the head.

          Show
          Benoit Chesneau added a comment - Thanks Adam, not sure why it has been committed at first... Anyway I've committed a new change : https://github.com/benoitc/couchdb/commit/2df4351cbfb28f543876377ba82be30175a95fff which fix that issue. If this is ok i will merge the changes and commit it to the head.
          Hide
          Paul Joseph Davis added a comment -

          This could just be the diff being weird, but does that Makefile.am work? The indentation doesn't look correct for the targets. Could just be the diff but I thought I'd mention it just in case.

          Show
          Paul Joseph Davis added a comment - This could just be the diff being weird, but does that Makefile.am work? The indentation doesn't look correct for the targets. Could just be the diff but I thought I'd mention it just in case.
          Hide
          Benoit Chesneau added a comment -

          @davisp I fixed the indentation in https://github.com/benoitc/couchdb/commit/84f4e4179d29d8ffaf19ef846ae53cea3287ba87

          but makefile was working without.

          Show
          Benoit Chesneau added a comment - @davisp I fixed the indentation in https://github.com/benoitc/couchdb/commit/84f4e4179d29d8ffaf19ef846ae53cea3287ba87 but makefile was working without.
          Hide
          Paul Joseph Davis added a comment -

          Looks like I should've clicked through to the file view. Apparently GitHub has decided to start representing tabs as two space indents in diffs. +1 from me then.

          Show
          Paul Joseph Davis added a comment - Looks like I should've clicked through to the file view. Apparently GitHub has decided to start representing tabs as two space indents in diffs. +1 from me then.
          Hide
          Adam Kocoloski added a comment -

          LGTM Benoit

          Show
          Adam Kocoloski added a comment - LGTM Benoit
          Hide
          Benoit Chesneau added a comment -

          rework previous patch and split it in two. The first one is moving parts around in the couch_replicator application folder and make sure tests pass and build work. The second is refactoring the couch_replicator application to put all the modules under the couch_replicator namespace.

          All tests pass.

          Let me know what you think about it. I would like it committed asap.

          Show
          Benoit Chesneau added a comment - rework previous patch and split it in two. The first one is moving parts around in the couch_replicator application folder and make sure tests pass and build work. The second is refactoring the couch_replicator application to put all the modules under the couch_replicator namespace. All tests pass. Let me know what you think about it. I would like it committed asap.
          Hide
          Benoit Chesneau added a comment -
          Show
          Benoit Chesneau added a comment - Full patch is available in my repo : https://github.com/benoitc/couchdb/compare/master...couch_replicator-2
          Hide
          Randall Leeds added a comment -

          Thanks, Benoit. Indentation in couch_replicator.app.src is inconsistent, but otherwise everything looks good. +1.

          Show
          Randall Leeds added a comment - Thanks, Benoit. Indentation in couch_replicator.app.src is inconsistent, but otherwise everything looks good. +1.
          Hide
          Benoit Chesneau added a comment -

          new version of 0002-refactor-couch_replicator.patch fixing inconsistency in indentation . spotted by randall , thanks!

          Show
          Benoit Chesneau added a comment - new version of 0002-refactor-couch_replicator.patch fixing inconsistency in indentation . spotted by randall , thanks!
          Hide
          Filipe Manana added a comment -

          Benoît, just tried out your github branch. All tests pass for me and it looks ok, so +1

          Looking at .gitignore changes, I see the following line added and I'm curious about it.

          +.sw

          What are the .sw files?

          Show
          Filipe Manana added a comment - Benoît, just tried out your github branch. All tests pass for me and it looks ok, so +1 Looking at .gitignore changes, I see the following line added and I'm curious about it. + .sw What are the .sw files?
          Hide
          Randall Leeds added a comment -

          Is there any reason not to ignore .* everywhere in the project instead?
          For example, I often blow away my .emacs (custom couchdb dev settings) and .emacs.deskop files doing a clean.
          Do we have any .* files that are legitimately included (aside from .gitignore)?

          A quick look (find . -iname ".*") doesn't show anything outside .git (aside from .gitignore) that's actually checked in.

          Show
          Randall Leeds added a comment - Is there any reason not to ignore .* everywhere in the project instead? For example, I often blow away my .emacs (custom couchdb dev settings) and .emacs.deskop files doing a clean. Do we have any .* files that are legitimately included (aside from .gitignore)? A quick look (find . -iname ".*") doesn't show anything outside .git (aside from .gitignore) that's actually checked in.
          Hide
          Paul Joseph Davis added a comment -

          I can't think of a reason not to.

          In other news, for vim users, you can use this in your vimrc to avoid the whole silly swap file issue in all projects. Found it a long time ago and have since forgotten about the dumb swap file thing completely.

          set directory=~/.backup/vim/swap
          set backupdir=~/.backup/vim
          set backup

          Show
          Paul Joseph Davis added a comment - I can't think of a reason not to. In other news, for vim users, you can use this in your vimrc to avoid the whole silly swap file issue in all projects. Found it a long time ago and have since forgotten about the dumb swap file thing completely. set directory=~/.backup/vim/swap set backupdir=~/.backup/vim set backup
          Hide
          Benoit Chesneau added a comment -

          @paul nice trick thanks.

          I will commit the patch later in the morning then. Thanks for the review.

          Show
          Benoit Chesneau added a comment - @paul nice trick thanks. I will commit the patch later in the morning then. Thanks for the review.
          Hide
          Benoit Chesneau added a comment -

          committed to the master.

          Show
          Benoit Chesneau added a comment - committed to the master.

            People

            • Assignee:
              Benoit Chesneau
              Reporter:
              Benoit Chesneau
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development