CouchDB
  1. CouchDB
  2. COUCHDB-852

CouchDB Windows - View building crashing fatally, unable to resume view building

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      Windows 7 - 32bit - Erlang R14A (erts-5.8) [source] [smp:4:4] [rq:4] [async-threads:0]

    • Skill Level:
      Regular Contributors Level (Easy to Medium)

      Description

      I have a DB with about 270,000 documents, inserted via a ruby script. Documents inserted fine, I get a working list of documents in the _all_docs view, can open an individual document up, etc.

      Tried to generate the appropriate views for the database by hitting a doc via futon, memory size/cpu usage stayed sane (35mb, 5-40% on a i5-750) for the first 180k documents. Then ballooned up to over 800mb and the view build eventually dies.

      [info] [<0.151.0>] checkpointing view update at seq 181904 for gsc_lt2 _design/mip

      Crash dump was written to: erl_crash.dump
      eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "old_heap").

      (Crash log for the couch/io installer, Mark Hammond's installer dies with a similar error message, but eheap_alloc reports "of type "heap"" instead)

      At which point CouchDB completely dies (expected, just mentioning it)

      Restarting CouchDB and trying to resume the view build fails fatally as well, with CouchDB reporting enomem errors in mochiweb. At this point, the already built view indexes appear to be totally unusable, I am unable to clean/compact the views, and I cannot find any way of restarting the build process other then by deleting the file holding the view (the md5sum-named file).

      Tested on two different harddisks, using both Windows installers (Mark Hammond + couch.io). Also further tested on request of benoitc on a vmware instance running ubuntu (NOT able to replicate the issue, view generation completes fine on the linux VM). Seems to be a Windows issue. VM instance had 1GB allocated to it, Machine has 4gb of RAM. Disk space on both tested harddisks remained > 20GB free at all times.

      For comparison purposes, memory usage on VM was stable at 5% of system RAM (5% of 1GB) throughout the entire view generation.

      The appropriate crash logs are:
      erl_crash.7z - for the first crash.
      couchdb.7z - containing two files:
      couch - start-firstcrash.log - CouchDB log files from startup, through view generation, till the first crash
      couch - start-secondcrash.log - CouchDB log files from startup, through view generation + first crash, then requerying views, till second (mochiweb) crash.

      1. couchdb.7z
        8 kB
        Lim Yue Chuan
      2. erl_crash.7z
        81 kB
        Lim Yue Chuan

        Issue Links

          Activity

          Lim Yue Chuan created issue -
          Lim Yue Chuan made changes -
          Field Original Value New Value
          Attachment erl_crash.7z [ 12451742 ]
          Attachment couchdb.7z [ 12451743 ]
          Lim Yue Chuan made changes -
          Description I have a DB with about 270,000 documents, inserted via a ruby script. Documents inserted fine, I get a working list of documents in the _all_docs view, can open an individual document up, etc.

          Tried to generate the appropriate views for the database by hitting a doc via futon, memory size/cpu usage stayed sane (35mb, 5-40% on a i5-750) for the first 180k documents. Then ballooned up to over 800mb and the view build eventually dies.

          [info] [<0.151.0>] checkpointing view update at seq 181904 for gsc_lt2 _design/mip

          Crash dump was written to: erl_crash.dump
          eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "old_heap").

          (Crash log for the couch/io installer, Mark Hammond's installer dies with a similar error message, but eheap_alloc reports "of type "heap"" instead)

          At which point CouchDB completely dies (expected, just mentioning it)

          Restarting CouchDB and trying to resume the view build fails fatally as well, with CouchDB reporting enomem errors in mochiweb. At this point, the already built view indexes appear to be totally unusable, I am unable to clean/compact the views, and I cannot find any way of restarting the build process other then by deleting the file holding the view (the md5sum-named file).

          Tested on two different harddisks, using both Windows installers (Mark Hammond + couch.io). Also further tested on request of benoitc on a vmware instance running ubuntu (NOT able to replicate the issue, view generation completes fine on the linux VM). Seems to be a Windows issue. VM instance had 1GB allocated to it, Machine has 4gb of RAM. Disk space on both tested harddisks remained > 20GB free at all times.

          For comparison purposes, memory usage on VM was stable at 5% of system RAM (5% of 1GB) throughout the entire view generation.

          The appropriate crash logs are:
          erl_crash.7z - for the first crash.
          couchdb.7z - containing two files:
            couch - start-firstcrash.log - CouchDB log files from startup, through view generation, till the first crash
            couch - start-secondcrash.log - CouchDB log files from startup, through view generation + first crash, then requerying views, till second (mochiweb) crash.

          Logs at:
          http://fall.purplelatte.com/couchdb/couchdb.7z
          http://fall.purplelatte.com/couchdb/erl_crash.7z
          I have a DB with about 270,000 documents, inserted via a ruby script. Documents inserted fine, I get a working list of documents in the _all_docs view, can open an individual document up, etc.

          Tried to generate the appropriate views for the database by hitting a doc via futon, memory size/cpu usage stayed sane (35mb, 5-40% on a i5-750) for the first 180k documents. Then ballooned up to over 800mb and the view build eventually dies.

          [info] [<0.151.0>] checkpointing view update at seq 181904 for gsc_lt2 _design/mip

          Crash dump was written to: erl_crash.dump
          eheap_alloc: Cannot allocate 373662860 bytes of memory (of type "old_heap").

          (Crash log for the couch/io installer, Mark Hammond's installer dies with a similar error message, but eheap_alloc reports "of type "heap"" instead)

          At which point CouchDB completely dies (expected, just mentioning it)

          Restarting CouchDB and trying to resume the view build fails fatally as well, with CouchDB reporting enomem errors in mochiweb. At this point, the already built view indexes appear to be totally unusable, I am unable to clean/compact the views, and I cannot find any way of restarting the build process other then by deleting the file holding the view (the md5sum-named file).

          Tested on two different harddisks, using both Windows installers (Mark Hammond + couch.io). Also further tested on request of benoitc on a vmware instance running ubuntu (NOT able to replicate the issue, view generation completes fine on the linux VM). Seems to be a Windows issue. VM instance had 1GB allocated to it, Machine has 4gb of RAM. Disk space on both tested harddisks remained > 20GB free at all times.

          For comparison purposes, memory usage on VM was stable at 5% of system RAM (5% of 1GB) throughout the entire view generation.

          The appropriate crash logs are:
          erl_crash.7z - for the first crash.
          couchdb.7z - containing two files:
            couch - start-firstcrash.log - CouchDB log files from startup, through view generation, till the first crash
            couch - start-secondcrash.log - CouchDB log files from startup, through view generation + first crash, then requerying views, till second (mochiweb) crash.
          Hide
          Lim Yue Chuan added a comment - - edited

          Tried importing in a batch and building the views incrementally. Appears that the view generation crashes when the view file hits 4gb in size.

          benoitc: appears to be related to the user's 32bit windows

          edit: on NTFS, so not a max-file-size issue

          Show
          Lim Yue Chuan added a comment - - edited Tried importing in a batch and building the views incrementally. Appears that the view generation crashes when the view file hits 4gb in size. benoitc: appears to be related to the user's 32bit windows edit: on NTFS, so not a max-file-size issue
          Lim Yue Chuan made changes -
          Link This issue is cloned as COUCHDB-897 [ COUCHDB-897 ]
          Paul Joseph Davis made changes -
          Skill Level Regular Contributors Level (Easy to Medium)
          Hide
          Filipe Manana added a comment -

          This is a Windows Erlang OTP bug.

          A fix (patch) for this issue was already submitted to the Erlang OTP team by Juhani Ränkimies:

          See this thread: http://erlang.2086793.n4.nabble.com/Fix-appending-to-large-files-gt-4GB-on-Windows-td2829384.html

          You might want to grab a Windows installer maintained by Juhani:

          http://github.com/juranki/couchdb/downloads

          Show
          Filipe Manana added a comment - This is a Windows Erlang OTP bug. A fix (patch) for this issue was already submitted to the Erlang OTP team by Juhani Ränkimies: See this thread: http://erlang.2086793.n4.nabble.com/Fix-appending-to-large-files-gt-4GB-on-Windows-td2829384.html You might want to grab a Windows installer maintained by Juhani: http://github.com/juranki/couchdb/downloads
          Filipe Manana made changes -
          Status Open [ 1 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Filipe Manana made changes -
          Link This issue blocks COUCHDB-897 [ COUCHDB-897 ]
          Filipe Manana made changes -
          Link This issue duplicates COUCHDB-897 [ COUCHDB-897 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Lim Yue Chuan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development