CouchDB
  1. CouchDB
  2. COUCHDB-999

Detect view indexes with duplicates in them

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.2, 1.0.1
    • Fix Version/s: 1.0.2, 1.1
    • Component/s: Database Core
    • Labels:
      None

      Description

      In COUCHDB-968 we discovered and fixed a bug that could introduce duplicate documents in a database. Under certain circumstances these duplicate documents would introduce duplicate entries in view indexes.

      We can detect duplicates in the view group by looking for duplicate entries in the docid => view key btree. We can try to repair the view indexes by removing the extra keys from the views, or we can just report that duplicates exist in the view group and direct the user to rebuild that group. We might incorporate this code into the view compactor, or we might release it as a separate standalone tool.

      I think my personal preference is to include any repair functionality in the view compactor. A separate standalone tool that simply reports the presence of duplicates would also be a nice thing to have.

        Issue Links

          Activity

          Hide
          Filipe Manana added a comment -

          Thanks Mike. I just applied it to trunk and branch 1.1.x

          Show
          Filipe Manana added a comment - Thanks Mike. I just applied it to trunk and branch 1.1.x
          Hide
          Mike Leddy added a comment -

          Applied the patch and it works fine..... thanks

          [Thu, 16 Jun 2011 19:41:27 GMT] [error] [<0.3449.0>] Duplicates of document `1305824272F8088` detected in view group `_design/base`, d
          atabase `iris` - view rebuild, from scratch, is required
          ... snip ...

            • Last message in was {'EXIT',<0.1030.0>,
              Unknown macro: {view_duplicated_id,<<"1305824272F8088">>}

              }

            • When Server state == {file,{file_descriptor,prim_file,{#Port<0.2781>,60}},
              3024207926}
            • Reason for termination ==
            • {view_duplicated_id,<<"1305824272F8088">>}
          Show
          Mike Leddy added a comment - Applied the patch and it works fine..... thanks [Thu, 16 Jun 2011 19:41:27 GMT] [error] [<0.3449.0>] Duplicates of document `1305824272F8088` detected in view group `_design/base`, d atabase `iris` - view rebuild, from scratch, is required ... snip ... Last message in was {'EXIT',<0.1030.0>, Unknown macro: {view_duplicated_id,<<"1305824272F8088">>} } When Server state == {file,{file_descriptor,prim_file,{#Port<0.2781>,60}}, 3024207926} Reason for termination == {view_duplicated_id,<<"1305824272F8088">>}
          Hide
          Filipe Manana added a comment -

          Mike do you want to try out this one liner patch:

          http://friendpaste.com/3rFccubDlgZNuFQ7hfpLNr

          and report back?

          thanks

          Show
          Filipe Manana added a comment - Mike do you want to try out this one liner patch: http://friendpaste.com/3rFccubDlgZNuFQ7hfpLNr and report back? thanks
          Hide
          Mike Leddy added a comment -

          I noticed that what appears in the couch.log is somewhat difficult to understand:

          [Thu, 16 Jun 2011 14:51:54 GMT] [error] [<0.886.0>] ** Generic server <0.886.0> terminating

            • Last message in was {'EXIT',<0.885.0>, [68,117,112,108,105,99,97,116,101,115,32,111, 102,32,"1307433591F4608",32,100,101,116,101, 99,116,101,100,32,105,110,32,"iris",32, "_design/admin",32,45,32,114,101,98,117,105, 108,100,32,114,101,113,117,105,114,101,100]}
            • When Server state == {file,{file_descriptor,prim_file,{#Port<0.2581>,56}},
              18063917287}
            • Reason for termination ==
            • [68,117,112,108,105,99,97,116,101,115,32,111,102,32,"1307433591F4608",32,
              100,101,116,101,99,116,101,100,32,105,110,32,"iris",32,"_design/admin",32,
              45,32,114,101,98,117,105,108,100,32,114,101,113,117,105,114,101,100]

          The exit message is not being shown as a normal 'string' in the log.

          Show
          Mike Leddy added a comment - I noticed that what appears in the couch.log is somewhat difficult to understand: [Thu, 16 Jun 2011 14:51:54 GMT] [error] [<0.886.0>] ** Generic server <0.886.0> terminating Last message in was {'EXIT',<0.885.0>, [68,117,112,108,105,99,97,116,101,115,32,111, 102,32,"1307433591F4608",32,100,101,116,101, 99,116,101,100,32,105,110,32,"iris",32, "_design/admin",32,45,32,114,101,98,117,105, 108,100,32,114,101,113,117,105,114,101,100]} When Server state == {file,{file_descriptor,prim_file,{#Port<0.2581>,56}}, 18063917287} Reason for termination == [68,117,112,108,105,99,97,116,101,115,32,111,102,32,"1307433591F4608",32, 100,101,116,101,99,116,101,100,32,105,110,32,"iris",32,"_design/admin",32, 45,32,114,101,98,117,105,108,100,32,114,101,113,117,105,114,101,100] The exit message is not being shown as a normal 'string' in the log.
          Hide
          Rachel Willmer added a comment -

          Is it not possible to do this in a way which self-heals?

          This is not a very scalable solution where there are a large number of namespaces which are affected.

          Rachel

          Show
          Rachel Willmer added a comment - Is it not possible to do this in a way which self-heals? This is not a very scalable solution where there are a large number of namespaces which are affected. Rachel
          Hide
          Adam Kocoloski added a comment -

          Done.

          Show
          Adam Kocoloski added a comment - Done.
          Hide
          Paul Joseph Davis added a comment -

          Looks good to me. +1 and lets get it in and get the latest versions out the door.

          Show
          Paul Joseph Davis added a comment - Looks good to me. +1 and lets get it in and get the latest versions out the door.
          Hide
          Adam Kocoloski added a comment -

          Here's an untested attempt at a patch to crash the view compactor if duplicates are encountered. The exit message in the logs will instruct the user to rebuild the affected view group.

          Show
          Adam Kocoloski added a comment - Here's an untested attempt at a patch to crash the view compactor if duplicates are encountered. The exit message in the logs will instruct the user to rebuild the affected view group.
          Hide
          Paul Joseph Davis added a comment -

          My preference would be for a view compactor that detects and reports the error without attempting to repair the index in place as that's the least amount of code.

          Show
          Paul Joseph Davis added a comment - My preference would be for a view compactor that detects and reports the error without attempting to repair the index in place as that's the least amount of code.

            People

            • Assignee:
              Unassigned
              Reporter:
              Adam Kocoloski
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development