CouchDB
  1. CouchDB
  2. COUCHDB-1167

Importing several documents and subsequently adding attachments fails and leads to crash report on DB server (see unit test)

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.0.2
    • Fix Version/s: None
    • Component/s: Database Core
    • Labels:
    • Environment:

      Windows 7 and Mac OS X, Python CouchDB API

    • Skill Level:
      Dont Know

      Description

      I just started to explore CouchDB and implemented a few Python unit tests.

      Perhaps I don't understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).

      (If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).

      Stefan

      --------------------------------------------

      (1)
      === CouchDB crash report ===
      Starting CouchDB...
      Eshell V5.8.1 (abort with ^G)
      1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
      1> Apache CouchDB has started. Time to relax.
      1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
      1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
      1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
      1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
      1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
      1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
      1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
      1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
      1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
      1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
      1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
      1> [error] [<0.131.0>] {error_report,<0.34.0>,
      {<0.131.0>,crash_report,
      [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},

      {pid,<0.131.0>}

      ,

      {registered_name,[]}

      ,
      {error_info,
      {error,

      {badmatch,ok},
      [{couch_httpd,handle_request_int,5},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]}},
      {ancestors, [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
      {messages,[]},
      {links,[<0.106.0>,#Port<0.1674>]},
      {dictionary,
      [{mochiweb_request_qs,[]},
      {jsonp,undefined},
      {mochiweb_request_cookie,[]}]},
      {trap_exit,false},
      {status,running},
      {heap_size,987},
      {stack_size,24},
      {reductions,13653}],
      []]}}
      1>
      =CRASH REPORT==== 17-May-2011::16:17:16 ===
      crasher:
      initial call: mochiweb_socket_server:acceptor_loop/1
      pid: <0.131.0>
      registered_name: []
      exception error: no match of right hand side value ok
      in function couch_httpd:handle_request_int/5
      in call from mochiweb_http:headers/5
      ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
      <0.35.0>]
      messages: []
      links: <0.106.0>,#Port<0.1674>
      dictionary: [{mochiweb_request_qs,[]},
      {jsonp,undefined},
      {mochiweb_request_cookie,[]}]
      trap_exit: false
      status: running
      heap_size: 987
      stack_size: 24
      reductions: 13653
      neighbours:
      1> [error] [<0.106.0>] {error_report,<0.34.0>,
      {<0.106.0>,std_error,
      {mochiweb_socket_server,235,{child_error,{badmatch,ok}

      }}}}
      1>
      =ERROR REPORT==== 17-May-2011::16:17:16 ===
      {mochiweb_socket_server,235,{child_error,

      {badmatch,ok}

      }}
      1>

      (2)
      === Python unittest output (of failed test) ===
      Finding files... done.
      Importing test modules ... done.

      <TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c'

      {'path': '.', 'name': 'doc1'}

      >
      <TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb'

      {'path': '.', 'name': 'doc2'}

      >
      <TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450'

      {'path': '.', 'name': 'doc3'}

      >
      Unexpected error: <type 'exceptions.AttributeError'>
      ======================================================================
      ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
      File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
      self.exportdbdocs(self.db, tmpexportfolder)
      File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
      self.exportdbdocattachment(db, row.key, root)
      File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
      f.write(attachment.read())
      AttributeError: 'NoneType' object has no attribute 'read'

      ----------------------------------------------------------------------
      Ran 1 test in 0.673s

      FAILED (errors=1)

      (3)
      === Python unit test code ===
      '''
      Created on May 17, 2011

      @author: sro
      '''
      import unittest
      import couchdb
      import filecmp
      import shutil
      import tempfile
      import sys
      import os
      from couchdb import Document
      from uuid import uuid4
      from pprint import pprint

      class TestDocument(Document):
      def _init_(self):
      self['_id'] = uuid4().hex

      def getpath(self):
      return self['path']

      def setpath(self, value):
      self['path'] = value

      def getname(self):
      return self['name']

      def setname(self, value):
      self['name'] = value

      class TestCouchDB(unittest.TestCase):

      def setUp(self):

      1. Change according to environment
        serverurl = "http://admin:admin@localhost:5984"
        dbname = "testdb"
        self.db = self.initdb(serverurl, dbname)

      def initdb(self, serverurl, dbname):
      couch = couchdb.Server(serverurl)

      1. Create empty database for tests
        try:
        couch.delete(dbname);
        except couchdb.http.ResourceNotFound:
        print("DB did not exist")
        finally:
        couch.create(dbname);
        db = couch[dbname]
        return db

      def exportdbdocs(self, db, root):
      map_fun = '''function(doc)

      { emit(doc._id, null); }

      '''
      for row in db.query(map_fun):
      self.exportdbdocattachment(db, row.key, root)

      def exportdbdocattachment(self, db, docid, root):
      doc = db.get(docid)
      attachment = db.get_attachment(doc.id, doc['name'])

      targetfolder = os.path.join(root, doc['path'])
      if not os.path.exists(targetfolder):
      os.makedirs(targetfolder)
      exportpath = os.path.join(targetfolder, doc['name'])

      f = open(exportpath, 'w')
      f.write(attachment.read())
      attachment.close()
      f.close

      return exportpath

      def createtestfile(self, path, size):
      f = open(path, 'w')
      f.write('\xff' * size)
      f.close()
      return f

      def createfilecollection(self, tmpfolder):
      path = []

      path.append(os.path.join(tmpfolder, 'doc1'))
      self.createtestfile(path[0], 100)

      path.append(os.path.join(tmpfolder, 'doc2'))
      self.createtestfile(path[1], 100)

      path.append(os.path.join(tmpfolder, 'doc3'))
      self.createtestfile(path[2], 100)

      return path

      def test_insertDocsAndAttachmentsInASingleLoop(self):

      1. This version of the test works
        try:
        docs = []
        tmpfolder = tempfile.mkdtemp()
        tmpexportfolder = tmpfolder + '-export'
      1. Here we first create all docs on the DB and then add attachments to them
        paths = self.createfilecollection(tmpfolder)
        for i in range(0, len(paths)):
      2. We append information about the initial attachment location
        relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
        filename = os.path.basename(paths[i])

      doc = TestDocument()
      doc.setpath(relpath)
      doc.setname(filename)
      docs.append(doc)

      self.db.save(doc)

      fp = open(paths[i])
      pprint(doc)
      self.db.put_attachment(doc, fp)
      fp.close()

      1. We export the attachments again and compate them to the originals
        self.exportdbdocs(self.db, tmpexportfolder)
        self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

      except:
      print "Unexpected error:", sys.exc_info()[0]
      raise

      finally:
      shutil.rmtree(tmpfolder)
      shutil.rmtree(tmpexportfolder)

      def test_insertDocsThenAttachments(self):

      1. This version of the test crashes CouchDB
        try:
        docs = []
        tmpfolder = tempfile.mkdtemp()
        tmpexportfolder = tmpfolder + '-export'
      1. Here we first create all docs on the DB and then add attachments to them
        paths = self.createfilecollection(tmpfolder)
        for i in range(0, len(paths)):
      2. We append information about the initial attachment location
        relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
        filename = os.path.basename(paths[i])

      doc = TestDocument()
      doc.setpath(relpath)
      doc.setname(filename)
      docs.append(doc)

      self.db.save(doc)

      1. Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
      2. self.db.update(docs)

      for i in range(0, len(paths)):
      doc = docs[i]
      fp = open(paths[i])
      pprint(doc)
      self.db.put_attachment(doc, fp)
      fp.close()

      1. We export the attachments and compare them to the originals
        self.exportdbdocs(self.db, tmpexportfolder)
        self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

      except:
      print "Unexpected error:", sys.exc_info()[0]
      raise

      finally:
      shutil.rmtree(tmpfolder)
      shutil.rmtree(tmpexportfolder)

      if _name_ == '_main_':
      suite = unittest.TestSuite()
      suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
      suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
      unittest.TextTestRunner().run(suite)

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Stefan Roth
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development