[COUCHDB-1167] Importing several documents and subsequently adding attachments fails and leads to crash report on DB server (see unit test)

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 1.0.2
• Component/s: Database Core
• Labels: attachments crash

Environment

Windows 7 and Mac OS X, Python CouchDB API

• Skill Level: Dont Know
• Assignee: Unassigned
• Reporter: Stefan Roth
• Created: 17/May/11 14:34
• Updated: 02/Sep/18 06:33
• Resolved: 02/Sep/18 06:33

Description

I just started to explore CouchDB and implemented a few Python unit tests.

Perhaps I don't understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).

(If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).

Stefan

--------------------------------------------

(1)
=== CouchDB crash report ===
Starting CouchDB...
Eshell V5.8.1 (abort with ^G)
1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
1> Apache CouchDB has started. Time to relax.
1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
1> [error] [<0.131.0>] {error_report,<0.34.0>,
{<0.131.0>,crash_report,
[[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},

{pid,<0.131.0>}

,

{registered_name,[]}

,
{error_info,
{error,

{badmatch,ok},
[{couch_httpd,handle_request_int,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}},
{ancestors, [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
{messages,[]},
{links,[<0.106.0>,#Port<0.1674>]},
{dictionary,
[{mochiweb_request_qs,[]},
{jsonp,undefined},
{mochiweb_request_cookie,[]}]},
{trap_exit,false},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,13653}],
[]]}}
1>
=CRASH REPORT==== 17-May-2011::16:17:16 ===
crasher:
initial call: mochiweb_socket_server:acceptor_loop/1
pid: <0.131.0>
registered_name: []
exception error: no match of right hand side value ok
in function couch_httpd:handle_request_int/5
in call from mochiweb_http:headers/5
ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
<0.35.0>]
messages: []
links: <0.106.0>,#Port<0.1674>
dictionary: [{mochiweb_request_qs,[]},
{jsonp,undefined},
{mochiweb_request_cookie,[]}]
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 13653
neighbours:
1> [error] [<0.106.0>] {error_report,<0.34.0>,
{<0.106.0>,std_error,
{mochiweb_socket_server,235,{child_error,{badmatch,ok}

}}}}
1>
=ERROR REPORT==== 17-May-2011::16:17:16 ===
{mochiweb_socket_server,235,{child_error,

{badmatch,ok}

}}
1>

(2)
=== Python unittest output (of failed test) ===
Finding files... done.
Importing test modules ... done.

<TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c'

{'path': '.', 'name': 'doc1'}

>
<TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb'

{'path': '.', 'name': 'doc2'}

>
<TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450'

{'path': '.', 'name': 'doc3'}

>
Unexpected error: <type 'exceptions.AttributeError'>
======================================================================
ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
self.exportdbdocs(self.db, tmpexportfolder)
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
self.exportdbdocattachment(db, row.key, root)
File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
f.write(attachment.read())
AttributeError: 'NoneType' object has no attribute 'read'

----------------------------------------------------------------------
Ran 1 test in 0.673s

FAILED (errors=1)

(3)
=== Python unit test code ===
'''
Created on May 17, 2011

@author: sro
'''
import unittest
import couchdb
import filecmp
import shutil
import tempfile
import sys
import os
from couchdb import Document
from uuid import uuid4
from pprint import pprint

class TestDocument(Document):
def _init_(self):
self['_id'] = uuid4().hex

def getpath(self):
return self['path']

def setpath(self, value):
self['path'] = value

def getname(self):
return self['name']

def setname(self, value):
self['name'] = value

class TestCouchDB(unittest.TestCase):

def setUp(self):

1. Change according to environment
   serverurl = "http://admin:admin@localhost:5984"
   dbname = "testdb"
   self.db = self.initdb(serverurl, dbname)

def initdb(self, serverurl, dbname):
couch = couchdb.Server(serverurl)

1. Create empty database for tests
   try:
   couch.delete(dbname);
   except couchdb.http.ResourceNotFound:
   print("DB did not exist")
   finally:
   couch.create(dbname);
   db = couch[dbname]
   return db

def exportdbdocs(self, db, root):
map_fun = '''function(doc)

{ emit(doc._id, null); }

'''
for row in db.query(map_fun):
self.exportdbdocattachment(db, row.key, root)

def exportdbdocattachment(self, db, docid, root):
doc = db.get(docid)
attachment = db.get_attachment(doc.id, doc['name'])

targetfolder = os.path.join(root, doc['path'])
if not os.path.exists(targetfolder):
os.makedirs(targetfolder)
exportpath = os.path.join(targetfolder, doc['name'])

f = open(exportpath, 'w')
f.write(attachment.read())
attachment.close()
f.close

return exportpath

def createtestfile(self, path, size):
f = open(path, 'w')
f.write('\xff' * size)
f.close()
return f

def createfilecollection(self, tmpfolder):
path = []

path.append(os.path.join(tmpfolder, 'doc1'))
self.createtestfile(path[0], 100)

path.append(os.path.join(tmpfolder, 'doc2'))
self.createtestfile(path[1], 100)

path.append(os.path.join(tmpfolder, 'doc3'))
self.createtestfile(path[2], 100)

return path

def test_insertDocsAndAttachmentsInASingleLoop(self):

1. This version of the test works
   try:
   docs = []
   tmpfolder = tempfile.mkdtemp()
   tmpexportfolder = tmpfolder + '-export'

1. Here we first create all docs on the DB and then add attachments to them
   paths = self.createfilecollection(tmpfolder)
   for i in range(0, len(paths)):
2. We append information about the initial attachment location
   relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
   filename = os.path.basename(paths[i])

doc = TestDocument()
doc.setpath(relpath)
doc.setname(filename)
docs.append(doc)

self.db.save(doc)

fp = open(paths[i])
pprint(doc)
self.db.put_attachment(doc, fp)
fp.close()

1. We export the attachments again and compate them to the originals
   self.exportdbdocs(self.db, tmpexportfolder)
   self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

except:
print "Unexpected error:", sys.exc_info()[0]
raise

finally:
shutil.rmtree(tmpfolder)
shutil.rmtree(tmpexportfolder)

def test_insertDocsThenAttachments(self):

1. This version of the test crashes CouchDB
   try:
   docs = []
   tmpfolder = tempfile.mkdtemp()
   tmpexportfolder = tmpfolder + '-export'

1. Here we first create all docs on the DB and then add attachments to them
   paths = self.createfilecollection(tmpfolder)
   for i in range(0, len(paths)):
2. We append information about the initial attachment location
   relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
   filename = os.path.basename(paths[i])

doc = TestDocument()
doc.setpath(relpath)
doc.setname(filename)
docs.append(doc)

self.db.save(doc)

1. Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
2. self.db.update(docs)

for i in range(0, len(paths)):
doc = docs[i]
fp = open(paths[i])
pprint(doc)
self.db.put_attachment(doc, fp)
fp.close()

1. We export the attachments and compare them to the originals
   self.exportdbdocs(self.db, tmpexportfolder)
   self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

except:
print "Unexpected error:", sys.exc_info()[0]
raise

finally:
shutil.rmtree(tmpfolder)
shutil.rmtree(tmpexportfolder)

if _name_ == '_main_':
suite = unittest.TestSuite()
suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
unittest.TextTestRunner().run(suite)