[COUCHDB-2886] Concurrent Bulk Updates and Attachments yield Corruption

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Assignee: Unassigned
• Reporter: Steven Schaefer
• Created: 19/Nov/15 18:37
• Updated: 02/Sep/18 07:07
• Resolved: 02/Sep/18 07:07

Description

I work for Nexidia, and we're evaluating CouchDB for a new system we're developing. I was running some tests and in general trying to push it as hard as I could to help gain experience and set expectations, and I've encountered what appears to be a database corruption issue, when conflicting concurrent batch updates include attachments. I noticed results from one of my views made no sense, and after verifying it was up-to-date, tried running view compaction in an attempt to address that problem. It errored with this:

[info] [<0.24269.24>] Compaction started for db: bug idx: _design/dev
[info] [<0.24224.24>] 127.0.0.1 - - POST /bug/_compact/dev 202
[error] [<0.24269.24>] Duplicate docid `9255f05ad2939b5c487209321c000c79` detected in view group `_design/dev`, database `bug` - This view needs to be rebuilt.
[error] [<0.24251.24>] ** Generic server <0.24251.24> terminating

• • Last message in was {'EXIT',<0.24269.24>,
    
    Unknown macro: {view_duplicate_id, <<"9255f05ad2939b5c487209321c000c79">>}

    }

  • When Server state ==
    Unknown macro: {st,<0.24244.24>,couch_mrview_index,<0.24269.24>}
  • Reason for termination ==
  • unknown_info

Deleting and then rebuilding the view didn't help, after which view results were still incorrect, and re-running view compaction produced the same error, which suggests to me a more fundamental issue.

I narrowed it down to a combination of concurrent bulk updates with attachments, or at least I should say I can consistently reproduce when including attachments, but I could never reproduce without them. I have attached a smallish example that usually reproduces the issue on my system.

Notes for running:

• Depends on Java 8 and if compiling I set this up to use maven
• Program will create "bug" database in a Couch instance running at localhost:5984. An existing "bug" db will be deleted.
• Some variables may need to be altered depending on system specs, look for "NOTE:" in the code. Rebuilding should be as easy as (assuming maven is installed) running from wherever unzipped: mvn install
• run-test.bat provides the command to run it. Only adjusting the -cp parameter should be necessary for Linux (i.e. ';' --> ':' )
• Should run fairly quickly, after which "Bug reproduced" should be visible in the console, and the Couch log will have an error like above.

System Notes:

• Windows 7
• 2x Intel Xeon E5-2687W v3 @ 3.10 GHz - 10 cores + hyperthreading (40 logical processors)
• 32 GB ram (NUMA)
• CouchDB configuration is default except for db and view storage locations.
• CouchDB 1.6.1