CouchDB
  1. CouchDB
  2. COUCHDB-780

Don't block the updater process while compaction deletes old files

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.2, 0.11
    • Fix Version/s: 1.0
    • Component/s: Database Core
    • Labels:

      Description

      I have what I think is a simple patch I'll attach. I don't see any reason not to include it unless rename operations can be seriously slow on some filesystems (but I expect this is not the case).

      1. 0001-async-file-deletions.-COUCHDB-780.patch
        5 kB
        Adam Kocoloski
      2. async_compact_delete.patch
        0.7 kB
        Randall Leeds

        Issue Links

          Activity

          Hide
          Adam Kocoloski added a comment -

          Damien, thanks for pointing that out. Yes, the two patches are quite similar – the primary difference is that the Windows patch uses random names for the deleted files, presumably to ensure that repeated deletions of the same file don't result in an attempt to recreate a file (which I guess is illegal?)

          Randall, I'm not really too concerned about deleted files hanging around. We're careful to remove any .compact files, but that's because a .compact file would get used as the DB file after the main one was deleted. If for some reason file:delete fails after file:rename succeeds, the file would never be "undeleted". I'd prefer simply putting some note in the docs saying that .deleted (or whatever) files can be safely removed if they exist after a crash.

          Show
          Adam Kocoloski added a comment - Damien, thanks for pointing that out. Yes, the two patches are quite similar – the primary difference is that the Windows patch uses random names for the deleted files, presumably to ensure that repeated deletions of the same file don't result in an attempt to recreate a file (which I guess is illegal?) Randall, I'm not really too concerned about deleted files hanging around. We're careful to remove any .compact files, but that's because a .compact file would get used as the DB file after the main one was deleted. If for some reason file:delete fails after file:rename succeeds, the file would never be "undeleted". I'd prefer simply putting some note in the docs saying that .deleted (or whatever) files can be safely removed if they exist after a crash.
          Hide
          Damien Katz added a comment -

          Haven't looked at the patch, but this sounds similar to mark hammonds patch for fixing the windows file problems (which requires a yet unreleased version of Erlang I think). Maybe Marks patch also fixes this problem, or could with a little more work.

          https://issues.apache.org/jira/browse/COUCHDB-86

          Show
          Damien Katz added a comment - Haven't looked at the patch, but this sounds similar to mark hammonds patch for fixing the windows file problems (which requires a yet unreleased version of Erlang I think). Maybe Marks patch also fixes this problem, or could with a little more work. https://issues.apache.org/jira/browse/COUCHDB-86
          Hide
          Randall Leeds added a comment -

          I like it. My only concern is that we clean up .delete files somewhere if a crash left them around. I'm not sure where the best place to do this is. Probably sprinkled around wherever we open things. What do you think?

          Show
          Randall Leeds added a comment - I like it. My only concern is that we clean up .delete files somewhere if a crash left them around. I'm not sure where the best place to do this is. Probably sprinkled around wherever we open things. What do you think?
          Hide
          Adam Kocoloski added a comment -

          Hi Randall, I liked the concept, so I took it a step further. I've attached a patch to add a couch_file:delete/1 function which does a rename-and-spawn-delete like you wrote in couch_db_updater. I've changed the codebase to use this function exclusively, with the exception of couch_view:nuke_dir which uses the error code from file:delete to ascertain that it tried to delete a directory.

          What do you think?

          Show
          Adam Kocoloski added a comment - Hi Randall, I liked the concept, so I took it a step further. I've attached a patch to add a couch_ file:delete/1 function which does a rename-and-spawn-delete like you wrote in couch_db_updater. I've changed the codebase to use this function exclusively, with the exception of couch_view:nuke_dir which uses the error code from file:delete to ascertain that it tried to delete a directory. What do you think?
          Hide
          Randall Leeds added a comment -

          I should try to compile first before I submit a patch.
          Fixe'd the syntax errors.
          [facepalm]

          Show
          Randall Leeds added a comment - I should try to compile first before I submit a patch. Fixe'd the syntax errors. [facepalm]
          Hide
          Randall Leeds added a comment -

          Instead of just deleting the old .couch file and then renaming the .compact file, move the .couch file out the way and delete it asynchronously. This patch should prevent slow deletion of huge files for filesystems using indirect blocks (like ext3) from blocking the updater process.

          Show
          Randall Leeds added a comment - Instead of just deleting the old .couch file and then renaming the .compact file, move the .couch file out the way and delete it asynchronously. This patch should prevent slow deletion of huge files for filesystems using indirect blocks (like ext3) from blocking the updater process.

            People

            • Assignee:
              Unassigned
              Reporter:
              Randall Leeds
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development