CouchDB
  1. CouchDB
  2. COUCHDB-1283

Impossible to compact view groups when number of active databases > max_dbs_open

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.1, 1.2
    • Component/s: None
    • Labels:
      None

      Description

      Mike Leddy recently reported this issue in the users mailing list:

      http://mail-archives.apache.org/mod_mbox/couchdb-user/201109.mbox/%3C1315949945.22123.22.camel@mike.loop.com.br%3E

      The attached patch is the simplest solution I can think of - keeping the database open until the view compaction finishes.
      The patch includes a test case.

      It will need to be updated after Paul's view index refactoring (COUCHDB-1270).

      1. couchdb-1283_trunk.patch
        10 kB
        Filipe Manana
      2. couchdb-1283_12x.patch
        12 kB
        Filipe Manana

        Activity

        Hide
        Filipe Manana added a comment -

        Paul,

        I made the necessary adaptions to the test case to use the new view modules. I also update couch_mrview_compactor, but this is not even close to what was done for 1.2.x (and 1.1.x) - namely we need to keep the database open for the eventual case of recompaction and when recompaction happens, reopen the database atomically via couch_db:reopen/1.

        Running this test, I get some error for couch_mrview_compactor when recompact happens.
        Hope you can give a hand here, as I'm not yet very familiar with the new view modules.

        Thanks

        Show
        Filipe Manana added a comment - Paul, I made the necessary adaptions to the test case to use the new view modules. I also update couch_mrview_compactor, but this is not even close to what was done for 1.2.x (and 1.1.x) - namely we need to keep the database open for the eventual case of recompaction and when recompaction happens, reopen the database atomically via couch_db:reopen/1. Running this test, I get some error for couch_mrview_compactor when recompact happens. Hope you can give a hand here, as I'm not yet very familiar with the new view modules. Thanks
        Hide
        Filipe Manana added a comment -

        I forgot to paste it yesterday, here's the error I get when a view recompact happens with trunk:

        $ ./test/etap/run -v test/etap/201-view-group-shutdown.t
        test/etap/201-view-group-shutdown.t ..

        1. Current time local 2011-09-16 00:10:22
        2. Using etap version "0.3.4"
          1..17
          Apache CouchDB 0.0.0 (LogLevel=info) is starting.
          Apache CouchDB has started. Time to relax.
          [info] [<0.2.0>] Apache CouchDB has started on http://127.0.0.1:58243/
        3. View group updated
          ok 1 - Spawned writer 1
          ok 2 - Spawned writer 2
          ok 3 - Writer 1 opened his database
          ok 4 - Writer 2 opened his database
        4. View group updated
          ok 5 - Spawned writer 3
          ok 6 - Writer 3 got {error, all_dbs_active}

          when opening his database
          ok 7 - Writer 1 still alive
          ok 8 - Writer 2 still alive
          ok 9 - Writer 3 still alive
          [info] [<0.179.0>] Recompacting index couch_test_view_group_shutdown _design/foo at 20001
          [error] [emulator] Error in process <0.190.0> with exit value: {undef,[{couch_index_updater,update,[couch_mrview_index,{mrst,<<16 bytes>>,<0.181.0>,<<30 bytes>>,<<11 bytes>>,<<10 bytes>>,[],

          {[]},[{mrview,0,0,0,[<<4 bytes>>,<<4 bytes>>,<<4 bytes>>,<<4 bytes>>,<<3 bytes>>],[],<<37 bytes>>,{btree,<0.181.0>,{29679751,{20000,[]},29237352},#Fun<couch_btree.3.126133433>,#Fun<couch_btree.4.37628535>,#Fun<couch_ejson_compare.less_json_ids.2>,#Fun<couch_mrview_util.8.13864802>,snappy},[]}],{btree,<0.181.0>,{384486,20000,384638},#Fun<couch_btree.3.126133433>,#Fun<couch_btree.4.37628535>,#Fun<couch_btree.5.9554535>,#Fun<couch_mrview_util.6.41372338>,snappy},20001,0,undefined,undefined,undefined,undefined,undefined,nil}]}]}

          =ERROR REPORT==== 16-Sep-2011::00:10:38 ===
          Error in process <0.190.0> with exit value: {undef,[{couch_index_updater,update,[couch_mrview_index,{mrst,<<16 bytes>>,<0.181.0>,<<30 bytes>>,<<11 bytes>>,<<10 bytes>>,[],{[]}

          ,[{mrview,0,0,0,[<<4 bytes>>,<<4 bytes>>,<<4 bytes>>,<<4 bytes>>,<<3 bytes>>],[],<<37 bytes>>,{btree,<0.181.0>,

          Unknown macro: {29679751,{20000,[]},29237352}

          ,#Fun<couch_btree.3.126133433>,#Fun<couch_btree.4.37628535>,#Fun<couch_ejson_compare.less_json_ids.2>,#Fun<couch_mrview_util.8.13864802>,snappy},[]}],

          Unknown macro: {btree,<0.181.0>,{384486,20000,384638},#Fun<couch_btree.3.126133433>,#Fun<couch_btree.4.37628535>,#Fun<couch_btree.5.9554535>,#Fun<couch_mrview_util.6.41372338>,snappy}

          ,20001,0,undefined,undefined,undefined,undefined,undefined,nil}]}]}

        couch_index_updater:update/2 doesn't exist, but there's a 3-arity version of it, however I'm not sure where to get the 3rd argument for it.

        Show
        Filipe Manana added a comment - I forgot to paste it yesterday, here's the error I get when a view recompact happens with trunk: $ ./test/etap/run -v test/etap/201-view-group-shutdown.t test/etap/201-view-group-shutdown.t .. Current time local 2011-09-16 00:10:22 Using etap version "0.3.4" 1..17 Apache CouchDB 0.0.0 (LogLevel=info) is starting. Apache CouchDB has started. Time to relax. [info] [<0.2.0>] Apache CouchDB has started on http://127.0.0.1:58243/ View group updated ok 1 - Spawned writer 1 ok 2 - Spawned writer 2 ok 3 - Writer 1 opened his database ok 4 - Writer 2 opened his database View group updated ok 5 - Spawned writer 3 ok 6 - Writer 3 got {error, all_dbs_active} when opening his database ok 7 - Writer 1 still alive ok 8 - Writer 2 still alive ok 9 - Writer 3 still alive [info] [<0.179.0>] Recompacting index couch_test_view_group_shutdown _design/foo at 20001 [error] [emulator] Error in process <0.190.0> with exit value: {undef,[{couch_index_updater,update,[couch_mrview_index,{mrst,<<16 bytes>>,<0.181.0>,<<30 bytes>>,<<11 bytes>>,<<10 bytes>>,[], {[]},[{mrview,0,0,0, [<<4 bytes>>,<<4 bytes>>,<<4 bytes>>,<<4 bytes>>,<<3 bytes>>] ,[],<<37 bytes>>,{btree,<0.181.0>,{29679751,{20000,[]},29237352},#Fun<couch_btree.3.126133433>,#Fun<couch_btree.4.37628535>,#Fun<couch_ejson_compare.less_json_ids.2>,#Fun<couch_mrview_util.8.13864802>,snappy},[]}],{btree,<0.181.0>,{384486,20000,384638},#Fun<couch_btree.3.126133433>,#Fun<couch_btree.4.37628535>,#Fun<couch_btree.5.9554535>,#Fun<couch_mrview_util.6.41372338>,snappy},20001,0,undefined,undefined,undefined,undefined,undefined,nil}]}]} =ERROR REPORT==== 16-Sep-2011::00:10:38 === Error in process <0.190.0> with exit value: {undef,[{couch_index_updater,update,[couch_mrview_index,{mrst,<<16 bytes>>,<0.181.0>,<<30 bytes>>,<<11 bytes>>,<<10 bytes>>,[],{[]} ,[{mrview,0,0,0, [<<4 bytes>>,<<4 bytes>>,<<4 bytes>>,<<4 bytes>>,<<3 bytes>>] ,[],<<37 bytes>>,{btree,<0.181.0>, Unknown macro: {29679751,{20000,[]},29237352} ,#Fun<couch_btree.3.126133433>,#Fun<couch_btree.4.37628535>,#Fun<couch_ejson_compare.less_json_ids.2>,#Fun<couch_mrview_util.8.13864802>,snappy},[]}], Unknown macro: {btree,<0.181.0>,{384486,20000,384638},#Fun<couch_btree.3.126133433>,#Fun<couch_btree.4.37628535>,#Fun<couch_btree.5.9554535>,#Fun<couch_mrview_util.6.41372338>,snappy} ,20001,0,undefined,undefined,undefined,undefined,undefined,nil}]}]} couch_index_updater:update/2 doesn't exist, but there's a 3-arity version of it, however I'm not sure where to get the 3rd argument for it.
        Hide
        Paul Joseph Davis added a comment -

        Should be fixed by 1173435

        Show
        Paul Joseph Davis added a comment - Should be fixed by 1173435

          People

          • Assignee:
            Paul Joseph Davis
            Reporter:
            Filipe Manana
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development