CouchDB
  1. CouchDB
  2. COUCHDB-1167

Importing several documents and subsequently adding attachments fails and leads to crash report on DB server (see unit test)

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.0.2
    • Fix Version/s: None
    • Component/s: Database Core
    • Labels:
    • Environment:

      Windows 7 and Mac OS X, Python CouchDB API

    • Skill Level:
      Dont Know

      Description

      I just started to explore CouchDB and implemented a few Python unit tests.

      Perhaps I don't understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).

      (If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).

      Stefan

      --------------------------------------------

      (1)
      === CouchDB crash report ===
      Starting CouchDB...
      Eshell V5.8.1 (abort with ^G)
      1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
      1> Apache CouchDB has started. Time to relax.
      1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
      1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
      1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
      1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
      1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
      1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
      1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
      1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
      1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
      1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
      1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
      1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
      1> [error] [<0.131.0>] {error_report,<0.34.0>,
      {<0.131.0>,crash_report,
      [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},

      {pid,<0.131.0>}

      ,

      {registered_name,[]}

      ,
      {error_info,
      {error,

      {badmatch,ok},
      [{couch_httpd,handle_request_int,5},
      {mochiweb_http,headers,5},
      {proc_lib,init_p_do_apply,3}]}},
      {ancestors, [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
      {messages,[]},
      {links,[<0.106.0>,#Port<0.1674>]},
      {dictionary,
      [{mochiweb_request_qs,[]},
      {jsonp,undefined},
      {mochiweb_request_cookie,[]}]},
      {trap_exit,false},
      {status,running},
      {heap_size,987},
      {stack_size,24},
      {reductions,13653}],
      []]}}
      1>
      =CRASH REPORT==== 17-May-2011::16:17:16 ===
      crasher:
      initial call: mochiweb_socket_server:acceptor_loop/1
      pid: <0.131.0>
      registered_name: []
      exception error: no match of right hand side value ok
      in function couch_httpd:handle_request_int/5
      in call from mochiweb_http:headers/5
      ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
      <0.35.0>]
      messages: []
      links: <0.106.0>,#Port<0.1674>
      dictionary: [{mochiweb_request_qs,[]},
      {jsonp,undefined},
      {mochiweb_request_cookie,[]}]
      trap_exit: false
      status: running
      heap_size: 987
      stack_size: 24
      reductions: 13653
      neighbours:
      1> [error] [<0.106.0>] {error_report,<0.34.0>,
      {<0.106.0>,std_error,
      {mochiweb_socket_server,235,{child_error,{badmatch,ok}

      }}}}
      1>
      =ERROR REPORT==== 17-May-2011::16:17:16 ===
      {mochiweb_socket_server,235,{child_error,

      {badmatch,ok}

      }}
      1>

      (2)
      === Python unittest output (of failed test) ===
      Finding files... done.
      Importing test modules ... done.

      <TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c'

      {'path': '.', 'name': 'doc1'}

      >
      <TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb'

      {'path': '.', 'name': 'doc2'}

      >
      <TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450'

      {'path': '.', 'name': 'doc3'}

      >
      Unexpected error: <type 'exceptions.AttributeError'>
      ======================================================================
      ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
      File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
      self.exportdbdocs(self.db, tmpexportfolder)
      File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
      self.exportdbdocattachment(db, row.key, root)
      File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
      f.write(attachment.read())
      AttributeError: 'NoneType' object has no attribute 'read'

      ----------------------------------------------------------------------
      Ran 1 test in 0.673s

      FAILED (errors=1)

      (3)
      === Python unit test code ===
      '''
      Created on May 17, 2011

      @author: sro
      '''
      import unittest
      import couchdb
      import filecmp
      import shutil
      import tempfile
      import sys
      import os
      from couchdb import Document
      from uuid import uuid4
      from pprint import pprint

      class TestDocument(Document):
      def _init_(self):
      self['_id'] = uuid4().hex

      def getpath(self):
      return self['path']

      def setpath(self, value):
      self['path'] = value

      def getname(self):
      return self['name']

      def setname(self, value):
      self['name'] = value

      class TestCouchDB(unittest.TestCase):

      def setUp(self):

      1. Change according to environment
        serverurl = "http://admin:admin@localhost:5984"
        dbname = "testdb"
        self.db = self.initdb(serverurl, dbname)

      def initdb(self, serverurl, dbname):
      couch = couchdb.Server(serverurl)

      1. Create empty database for tests
        try:
        couch.delete(dbname);
        except couchdb.http.ResourceNotFound:
        print("DB did not exist")
        finally:
        couch.create(dbname);
        db = couch[dbname]
        return db

      def exportdbdocs(self, db, root):
      map_fun = '''function(doc)

      { emit(doc._id, null); }

      '''
      for row in db.query(map_fun):
      self.exportdbdocattachment(db, row.key, root)

      def exportdbdocattachment(self, db, docid, root):
      doc = db.get(docid)
      attachment = db.get_attachment(doc.id, doc['name'])

      targetfolder = os.path.join(root, doc['path'])
      if not os.path.exists(targetfolder):
      os.makedirs(targetfolder)
      exportpath = os.path.join(targetfolder, doc['name'])

      f = open(exportpath, 'w')
      f.write(attachment.read())
      attachment.close()
      f.close

      return exportpath

      def createtestfile(self, path, size):
      f = open(path, 'w')
      f.write('\xff' * size)
      f.close()
      return f

      def createfilecollection(self, tmpfolder):
      path = []

      path.append(os.path.join(tmpfolder, 'doc1'))
      self.createtestfile(path[0], 100)

      path.append(os.path.join(tmpfolder, 'doc2'))
      self.createtestfile(path[1], 100)

      path.append(os.path.join(tmpfolder, 'doc3'))
      self.createtestfile(path[2], 100)

      return path

      def test_insertDocsAndAttachmentsInASingleLoop(self):

      1. This version of the test works
        try:
        docs = []
        tmpfolder = tempfile.mkdtemp()
        tmpexportfolder = tmpfolder + '-export'
      1. Here we first create all docs on the DB and then add attachments to them
        paths = self.createfilecollection(tmpfolder)
        for i in range(0, len(paths)):
      2. We append information about the initial attachment location
        relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
        filename = os.path.basename(paths[i])

      doc = TestDocument()
      doc.setpath(relpath)
      doc.setname(filename)
      docs.append(doc)

      self.db.save(doc)

      fp = open(paths[i])
      pprint(doc)
      self.db.put_attachment(doc, fp)
      fp.close()

      1. We export the attachments again and compate them to the originals
        self.exportdbdocs(self.db, tmpexportfolder)
        self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

      except:
      print "Unexpected error:", sys.exc_info()[0]
      raise

      finally:
      shutil.rmtree(tmpfolder)
      shutil.rmtree(tmpexportfolder)

      def test_insertDocsThenAttachments(self):

      1. This version of the test crashes CouchDB
        try:
        docs = []
        tmpfolder = tempfile.mkdtemp()
        tmpexportfolder = tmpfolder + '-export'
      1. Here we first create all docs on the DB and then add attachments to them
        paths = self.createfilecollection(tmpfolder)
        for i in range(0, len(paths)):
      2. We append information about the initial attachment location
        relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
        filename = os.path.basename(paths[i])

      doc = TestDocument()
      doc.setpath(relpath)
      doc.setname(filename)
      docs.append(doc)

      self.db.save(doc)

      1. Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
      2. self.db.update(docs)

      for i in range(0, len(paths)):
      doc = docs[i]
      fp = open(paths[i])
      pprint(doc)
      self.db.put_attachment(doc, fp)
      fp.close()

      1. We export the attachments and compare them to the originals
        self.exportdbdocs(self.db, tmpexportfolder)
        self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

      except:
      print "Unexpected error:", sys.exc_info()[0]
      raise

      finally:
      shutil.rmtree(tmpfolder)
      shutil.rmtree(tmpexportfolder)

      if _name_ == '_main_':
      suite = unittest.TestSuite()
      suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
      suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
      unittest.TextTestRunner().run(suite)

        Activity

        Stefan Roth created issue -
        Stefan Roth made changes -
        Field Original Value New Value
        Summary Importing several document and adding attachments subsequently fails, and leads to crash report on server (see unit test) Importing several documents and subsequently adding attachments fails and leads to crash report on DB server (see unit test)
        Description I just started to explore CouchDB and implemented a few Python unit tests.

        Perhaps I understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).

        (If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).

        Stefan

        --------------------------------------------

        (1)
        === CouchDB crash report ===
        Starting CouchDB...
        Eshell V5.8.1 (abort with ^G)
        1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
        1> Apache CouchDB has started. Time to relax.
        1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
        1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
        1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
        1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
        1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
        1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
        1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
        1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
        1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
        1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
        1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
        1> [error] [<0.131.0>] {error_report,<0.34.0>,
            {<0.131.0>,crash_report,
             [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
               {pid,<0.131.0>},
               {registered_name,[]},
               {error_info,
                   {error,
                       {badmatch,ok},
                       [{couch_httpd,handle_request_int,5},
                        {mochiweb_http,headers,5},
                        {proc_lib,init_p_do_apply,3}]}},
               {ancestors,
                   [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
               {messages,[]},
               {links,[<0.106.0>,#Port<0.1674>]},
               {dictionary,
                   [{mochiweb_request_qs,[]},
                    {jsonp,undefined},
                    {mochiweb_request_cookie,[]}]},
               {trap_exit,false},
               {status,running},
               {heap_size,987},
               {stack_size,24},
               {reductions,13653}],
              []]}}
        1>
        =CRASH REPORT==== 17-May-2011::16:17:16 ===
          crasher:
            initial call: mochiweb_socket_server:acceptor_loop/1
            pid: <0.131.0>
            registered_name: []
            exception error: no match of right hand side value ok
              in function couch_httpd:handle_request_int/5
              in call from mochiweb_http:headers/5
            ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
                          <0.35.0>]
            messages: []
            links: [<0.106.0>,#Port<0.1674>]
            dictionary: [{mochiweb_request_qs,[]},
                          {jsonp,undefined},
                          {mochiweb_request_cookie,[]}]
            trap_exit: false
            status: running
            heap_size: 987
            stack_size: 24
            reductions: 13653
          neighbours:
        1> [error] [<0.106.0>] {error_report,<0.34.0>,
                      {<0.106.0>,std_error,
                       {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
        1>
        =ERROR REPORT==== 17-May-2011::16:17:16 ===
        {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
        1>

        (2)
        === Python unittest output (of failed test) ===
        Finding files... done.
        Importing test modules ... done.

        <TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c' {'path': '.', 'name': 'doc1'}>
        <TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb' {'path': '.', 'name': 'doc2'}>
        <TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450' {'path': '.', 'name': 'doc3'}>
        Unexpected error: <type 'exceptions.AttributeError'>
        ======================================================================
        ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
            self.exportdbdocs(self.db, tmpexportfolder)
          File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
            self.exportdbdocattachment(db, row.key, root)
          File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
            f.write(attachment.read())
        AttributeError: 'NoneType' object has no attribute 'read'

        ----------------------------------------------------------------------
        Ran 1 test in 0.673s

        FAILED (errors=1)


        (3)
        === Python unit test code ===
        '''
        Created on May 17, 2011

        @author: sro
        '''
        import unittest
        import couchdb
        import filecmp
        import shutil
        import tempfile
        import sys
        import os
        from couchdb import Document
        from uuid import uuid4
        from pprint import pprint

        class TestDocument(Document):
            def __init__(self):
                self['_id'] = uuid4().hex
                
            def getpath(self):
                return self['path']

            def setpath(self, value):
                self['path'] = value
            
            def getname(self):
                return self['name']
            
            def setname(self, value):
                self['name'] = value

        class TestCouchDB(unittest.TestCase):

            def setUp(self):
                # Change according to environment
                serverurl = "http://admin:admin@localhost:5984"
                dbname = "testdb"
                self.db = self.initdb(serverurl, dbname)
                
            def initdb(self, serverurl, dbname):
                couch = couchdb.Server(serverurl)
                # Create empty database for tests
                try:
                    couch.delete(dbname);
                except couchdb.http.ResourceNotFound:
                    print("DB did not exist")
                finally:
                    couch.create(dbname);
                db = couch[dbname]
                return db

            def exportdbdocs(self, db, root):
                map_fun = '''function(doc) {
                    emit(doc._id, null);
                    }'''
                for row in db.query(map_fun):
                    self.exportdbdocattachment(db, row.key, root)

            def exportdbdocattachment(self, db, docid, root):
                doc = db.get(docid)
                attachment = db.get_attachment(doc.id, doc['name'])
                
                targetfolder = os.path.join(root, doc['path'])
                if not os.path.exists(targetfolder):
                    os.makedirs(targetfolder)
                exportpath = os.path.join(targetfolder, doc['name'])
                  
                f = open(exportpath, 'w')
                f.write(attachment.read())
                attachment.close()
                f.close
                
                return exportpath

            def createtestfile(self, path, size):
                f = open(path, 'w')
                f.write('\xff' * size)
                f.close()
                return f
                
            def createfilecollection(self, tmpfolder):
                path = []
                
                path.append(os.path.join(tmpfolder, 'doc1'))
                self.createtestfile(path[0], 100)
                
                path.append(os.path.join(tmpfolder, 'doc2'))
                self.createtestfile(path[1], 100)
                
                path.append(os.path.join(tmpfolder, 'doc3'))
                self.createtestfile(path[2], 100)
                
                return path
             
            def test_insertDocsAndAttachmentsInASingleLoop(self):
                # This version of the test works
                try:
                    docs = []
                    tmpfolder = tempfile.mkdtemp()
                    tmpexportfolder = tmpfolder + '-export'
                    
                    # Here we first create all docs on the DB and then add attachments to them
                    paths = self.createfilecollection(tmpfolder)
                    for i in range(0, len(paths)):
                        # We append information about the initial attachment location
                        relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
                        filename = os.path.basename(paths[i])
                        
                        doc = TestDocument()
                        doc.setpath(relpath)
                        doc.setname(filename)
                        docs.append(doc)
                        
                        self.db.save(doc)

                        fp = open(paths[i])
                        pprint(doc)
                        self.db.put_attachment(doc, fp)
                        fp.close()
                    
                    # We export the attachments again and compate them to the originals
                    self.exportdbdocs(self.db, tmpexportfolder)
                    self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

                except:
                    print "Unexpected error:", sys.exc_info()[0]
                    raise
                
                finally:
                    shutil.rmtree(tmpfolder)
                    shutil.rmtree(tmpexportfolder)

            def test_insertDocsThenAttachments(self):
                # This version of the test crashes CouchDB
                try:
                    docs = []
                    tmpfolder = tempfile.mkdtemp()
                    tmpexportfolder = tmpfolder + '-export'
                    
                    # Here we first create all docs on the DB and then add attachments to them
                    paths = self.createfilecollection(tmpfolder)
                    for i in range(0, len(paths)):
                        # We append information about the initial attachment location
                        relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
                        filename = os.path.basename(paths[i])
                        
                        doc = TestDocument()
                        doc.setpath(relpath)
                        doc.setname(filename)
                        docs.append(doc)
                        
                        self.db.save(doc)

                    # Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
                    # self.db.update(docs)
                    
                    for i in range(0, len(paths)):
                        doc = docs[i]
                        fp = open(paths[i])
                        pprint(doc)
                        self.db.put_attachment(doc, fp)
                        fp.close()
                    
                    # We export the attachments and compare them to the originals
                    self.exportdbdocs(self.db, tmpexportfolder)
                    self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

                except:
                    print "Unexpected error:", sys.exc_info()[0]
                    raise
                
                finally:
                    shutil.rmtree(tmpfolder)
                    shutil.rmtree(tmpexportfolder)
                    
        if __name__ == '__main__':
            suite = unittest.TestSuite()
            suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
            suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
            unittest.TextTestRunner().run(suite)
                
        I just started to explore CouchDB and implemented a few Python unit tests.

        Perhaps I don't understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)).

        (If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).

        Stefan

        --------------------------------------------

        (1)
        === CouchDB crash report ===
        Starting CouchDB...
        Eshell V5.8.1 (abort with ^G)
        1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
        1> Apache CouchDB has started. Time to relax.
        1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
        1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
        1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
        1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
        1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
        1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
        1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
        1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
        1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
        1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
        1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
        1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
        1> [error] [<0.131.0>] {error_report,<0.34.0>,
            {<0.131.0>,crash_report,
             [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
               {pid,<0.131.0>},
               {registered_name,[]},
               {error_info,
                   {error,
                       {badmatch,ok},
                       [{couch_httpd,handle_request_int,5},
                        {mochiweb_http,headers,5},
                        {proc_lib,init_p_do_apply,3}]}},
               {ancestors,
                   [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
               {messages,[]},
               {links,[<0.106.0>,#Port<0.1674>]},
               {dictionary,
                   [{mochiweb_request_qs,[]},
                    {jsonp,undefined},
                    {mochiweb_request_cookie,[]}]},
               {trap_exit,false},
               {status,running},
               {heap_size,987},
               {stack_size,24},
               {reductions,13653}],
              []]}}
        1>
        =CRASH REPORT==== 17-May-2011::16:17:16 ===
          crasher:
            initial call: mochiweb_socket_server:acceptor_loop/1
            pid: <0.131.0>
            registered_name: []
            exception error: no match of right hand side value ok
              in function couch_httpd:handle_request_int/5
              in call from mochiweb_http:headers/5
            ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
                          <0.35.0>]
            messages: []
            links: [<0.106.0>,#Port<0.1674>]
            dictionary: [{mochiweb_request_qs,[]},
                          {jsonp,undefined},
                          {mochiweb_request_cookie,[]}]
            trap_exit: false
            status: running
            heap_size: 987
            stack_size: 24
            reductions: 13653
          neighbours:
        1> [error] [<0.106.0>] {error_report,<0.34.0>,
                      {<0.106.0>,std_error,
                       {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
        1>
        =ERROR REPORT==== 17-May-2011::16:17:16 ===
        {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
        1>

        (2)
        === Python unittest output (of failed test) ===
        Finding files... done.
        Importing test modules ... done.

        <TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c' {'path': '.', 'name': 'doc1'}>
        <TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb' {'path': '.', 'name': 'doc2'}>
        <TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450' {'path': '.', 'name': 'doc3'}>
        Unexpected error: <type 'exceptions.AttributeError'>
        ======================================================================
        ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
            self.exportdbdocs(self.db, tmpexportfolder)
          File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
            self.exportdbdocattachment(db, row.key, root)
          File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
            f.write(attachment.read())
        AttributeError: 'NoneType' object has no attribute 'read'

        ----------------------------------------------------------------------
        Ran 1 test in 0.673s

        FAILED (errors=1)


        (3)
        === Python unit test code ===
        '''
        Created on May 17, 2011

        @author: sro
        '''
        import unittest
        import couchdb
        import filecmp
        import shutil
        import tempfile
        import sys
        import os
        from couchdb import Document
        from uuid import uuid4
        from pprint import pprint

        class TestDocument(Document):
            def __init__(self):
                self['_id'] = uuid4().hex
                
            def getpath(self):
                return self['path']

            def setpath(self, value):
                self['path'] = value
            
            def getname(self):
                return self['name']
            
            def setname(self, value):
                self['name'] = value

        class TestCouchDB(unittest.TestCase):

            def setUp(self):
                # Change according to environment
                serverurl = "http://admin:admin@localhost:5984"
                dbname = "testdb"
                self.db = self.initdb(serverurl, dbname)
                
            def initdb(self, serverurl, dbname):
                couch = couchdb.Server(serverurl)
                # Create empty database for tests
                try:
                    couch.delete(dbname);
                except couchdb.http.ResourceNotFound:
                    print("DB did not exist")
                finally:
                    couch.create(dbname);
                db = couch[dbname]
                return db

            def exportdbdocs(self, db, root):
                map_fun = '''function(doc) {
                    emit(doc._id, null);
                    }'''
                for row in db.query(map_fun):
                    self.exportdbdocattachment(db, row.key, root)

            def exportdbdocattachment(self, db, docid, root):
                doc = db.get(docid)
                attachment = db.get_attachment(doc.id, doc['name'])
                
                targetfolder = os.path.join(root, doc['path'])
                if not os.path.exists(targetfolder):
                    os.makedirs(targetfolder)
                exportpath = os.path.join(targetfolder, doc['name'])
                  
                f = open(exportpath, 'w')
                f.write(attachment.read())
                attachment.close()
                f.close
                
                return exportpath

            def createtestfile(self, path, size):
                f = open(path, 'w')
                f.write('\xff' * size)
                f.close()
                return f
                
            def createfilecollection(self, tmpfolder):
                path = []
                
                path.append(os.path.join(tmpfolder, 'doc1'))
                self.createtestfile(path[0], 100)
                
                path.append(os.path.join(tmpfolder, 'doc2'))
                self.createtestfile(path[1], 100)
                
                path.append(os.path.join(tmpfolder, 'doc3'))
                self.createtestfile(path[2], 100)
                
                return path
             
            def test_insertDocsAndAttachmentsInASingleLoop(self):
                # This version of the test works
                try:
                    docs = []
                    tmpfolder = tempfile.mkdtemp()
                    tmpexportfolder = tmpfolder + '-export'
                    
                    # Here we first create all docs on the DB and then add attachments to them
                    paths = self.createfilecollection(tmpfolder)
                    for i in range(0, len(paths)):
                        # We append information about the initial attachment location
                        relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
                        filename = os.path.basename(paths[i])
                        
                        doc = TestDocument()
                        doc.setpath(relpath)
                        doc.setname(filename)
                        docs.append(doc)
                        
                        self.db.save(doc)

                        fp = open(paths[i])
                        pprint(doc)
                        self.db.put_attachment(doc, fp)
                        fp.close()
                    
                    # We export the attachments again and compate them to the originals
                    self.exportdbdocs(self.db, tmpexportfolder)
                    self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

                except:
                    print "Unexpected error:", sys.exc_info()[0]
                    raise
                
                finally:
                    shutil.rmtree(tmpfolder)
                    shutil.rmtree(tmpexportfolder)

            def test_insertDocsThenAttachments(self):
                # This version of the test crashes CouchDB
                try:
                    docs = []
                    tmpfolder = tempfile.mkdtemp()
                    tmpexportfolder = tmpfolder + '-export'
                    
                    # Here we first create all docs on the DB and then add attachments to them
                    paths = self.createfilecollection(tmpfolder)
                    for i in range(0, len(paths)):
                        # We append information about the initial attachment location
                        relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)
                        filename = os.path.basename(paths[i])
                        
                        doc = TestDocument()
                        doc.setpath(relpath)
                        doc.setname(filename)
                        docs.append(doc)
                        
                        self.db.save(doc)

                    # Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
                    # self.db.update(docs)
                    
                    for i in range(0, len(paths)):
                        doc = docs[i]
                        fp = open(paths[i])
                        pprint(doc)
                        self.db.put_attachment(doc, fp)
                        fp.close()
                    
                    # We export the attachments and compare them to the originals
                    self.exportdbdocs(self.db, tmpexportfolder)
                    self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

                except:
                    print "Unexpected error:", sys.exc_info()[0]
                    raise
                
                finally:
                    shutil.rmtree(tmpfolder)
                    shutil.rmtree(tmpexportfolder)
                    
        if __name__ == '__main__':
            suite = unittest.TestSuite()
            suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
            suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
            unittest.TextTestRunner().run(suite)
                

          People

          • Assignee:
            Unassigned
            Reporter:
            Stefan Roth
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development