CouchDB
  1. CouchDB
  2. COUCHDB-780

Don't block the updater process while compaction deletes old files

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.2, 0.11
    • Fix Version/s: 1.0
    • Component/s: Database Core
    • Labels:

      Description

      I have what I think is a simple patch I'll attach. I don't see any reason not to include it unless rename operations can be seriously slow on some filesystems (but I expect this is not the case).

      1. async_compact_delete.patch
        0.7 kB
        Randall Leeds
      2. 0001-async-file-deletions.-COUCHDB-780.patch
        5 kB
        Adam Kocoloski

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          24d 18h 58m 1 Damien Katz 23/Jun/10 19:00
          Resolved Resolved Closed Closed
          3m 56s 1 Randall Leeds 23/Jun/10 19:04
          Randall Leeds made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Damien Katz made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 1.0 [ 12313209 ]
          Fix Version/s 1.1 [ 12314933 ]
          Resolution Fixed [ 1 ]
          Hide
          Adam Kocoloski added a comment -

          Damien, thanks for pointing that out. Yes, the two patches are quite similar – the primary difference is that the Windows patch uses random names for the deleted files, presumably to ensure that repeated deletions of the same file don't result in an attempt to recreate a file (which I guess is illegal?)

          Randall, I'm not really too concerned about deleted files hanging around. We're careful to remove any .compact files, but that's because a .compact file would get used as the DB file after the main one was deleted. If for some reason file:delete fails after file:rename succeeds, the file would never be "undeleted". I'd prefer simply putting some note in the docs saying that .deleted (or whatever) files can be safely removed if they exist after a crash.

          Show
          Adam Kocoloski added a comment - Damien, thanks for pointing that out. Yes, the two patches are quite similar – the primary difference is that the Windows patch uses random names for the deleted files, presumably to ensure that repeated deletions of the same file don't result in an attempt to recreate a file (which I guess is illegal?) Randall, I'm not really too concerned about deleted files hanging around. We're careful to remove any .compact files, but that's because a .compact file would get used as the DB file after the main one was deleted. If for some reason file:delete fails after file:rename succeeds, the file would never be "undeleted". I'd prefer simply putting some note in the docs saying that .deleted (or whatever) files can be safely removed if they exist after a crash.
          Adam Kocoloski made changes -
          Link This issue is related to COUCHDB-86 [ COUCHDB-86 ]
          Hide
          Damien Katz added a comment -

          Haven't looked at the patch, but this sounds similar to mark hammonds patch for fixing the windows file problems (which requires a yet unreleased version of Erlang I think). Maybe Marks patch also fixes this problem, or could with a little more work.

          https://issues.apache.org/jira/browse/COUCHDB-86

          Show
          Damien Katz added a comment - Haven't looked at the patch, but this sounds similar to mark hammonds patch for fixing the windows file problems (which requires a yet unreleased version of Erlang I think). Maybe Marks patch also fixes this problem, or could with a little more work. https://issues.apache.org/jira/browse/COUCHDB-86
          Hide
          Randall Leeds added a comment -

          I like it. My only concern is that we clean up .delete files somewhere if a crash left them around. I'm not sure where the best place to do this is. Probably sprinkled around wherever we open things. What do you think?

          Show
          Randall Leeds added a comment - I like it. My only concern is that we clean up .delete files somewhere if a crash left them around. I'm not sure where the best place to do this is. Probably sprinkled around wherever we open things. What do you think?
          Adam Kocoloski made changes -
          Hide
          Adam Kocoloski added a comment -

          Hi Randall, I liked the concept, so I took it a step further. I've attached a patch to add a couch_file:delete/1 function which does a rename-and-spawn-delete like you wrote in couch_db_updater. I've changed the codebase to use this function exclusively, with the exception of couch_view:nuke_dir which uses the error code from file:delete to ascertain that it tried to delete a directory.

          What do you think?

          Show
          Adam Kocoloski added a comment - Hi Randall, I liked the concept, so I took it a step further. I've attached a patch to add a couch_ file:delete/1 function which does a rename-and-spawn-delete like you wrote in couch_db_updater. I've changed the codebase to use this function exclusively, with the exception of couch_view:nuke_dir which uses the error code from file:delete to ascertain that it tried to delete a directory. What do you think?
          Randall Leeds made changes -
          Fix Version/s 1.1 [ 12314933 ]
          Fix Version/s 0.12 [ 12314314 ]
          Fix Version/s 0.11.1 [ 12314871 ]
          Fix Version/s 0.10.3 [ 12314957 ]
          Randall Leeds made changes -
          Attachment async_compact_delete.patch [ 12445867 ]
          Randall Leeds made changes -
          Attachment async_compact_delete.patch [ 12445862 ]
          Randall Leeds made changes -
          Attachment async_compact_delete.patch [ 12445866 ]
          Randall Leeds made changes -
          Attachment async_compact_delete.patch [ 12445866 ]
          Hide
          Randall Leeds added a comment -

          I should try to compile first before I submit a patch.
          Fixe'd the syntax errors.
          [facepalm]

          Show
          Randall Leeds added a comment - I should try to compile first before I submit a patch. Fixe'd the syntax errors. [facepalm]
          Randall Leeds made changes -
          Field Original Value New Value
          Attachment async_compact_delete.patch [ 12445862 ]
          Hide
          Randall Leeds added a comment -

          Instead of just deleting the old .couch file and then renaming the .compact file, move the .couch file out the way and delete it asynchronously. This patch should prevent slow deletion of huge files for filesystems using indirect blocks (like ext3) from blocking the updater process.

          Show
          Randall Leeds added a comment - Instead of just deleting the old .couch file and then renaming the .compact file, move the .couch file out the way and delete it asynchronously. This patch should prevent slow deletion of huge files for filesystems using indirect blocks (like ext3) from blocking the updater process.
          Randall Leeds created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Randall Leeds
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development